Yury V. Zaytsev

Who framed mkfs? or Winners never cheat and cheaters never win

2011-06-07T00:00:00-05:00

I/O Limits: Quest for increased drive capacity

My first Armstrad had a glorious 20M hard drive and now I sometimes feel dissatisfied about the capacity of a shiny new 2T RAID array. Some 1 000 000x increase in 15 years, in fact, does not that sound quite amazing when one thinks about it?

The storage devices used to expose the data in 512B large addressable blocks to the OS. However, as their capacity has been steadily increasing, it became clear that the overhead associated with each sector on the current 512 byte sector disks is becoming a limiting factor.

In the reality, apart from 512 bytes worth of data, each physical sector contains quite a bit of extra information, such as an error correction checksum, sync label, etc. In some cases, these overheads amounted up to 15% of the usable storage space. For reasonably large drives of 2T and more we might be speaking about some 300G lost due to the layout format inefficiency.

Therefore, as storage vendors quickly realized, one way to obtain a major increase in capacity is to reduce the overheads associated with storing each physical sector on disk. That is why most of the modern hard drives operate with 4K sectors internally (physical_block_size), while exposing 512B sectors (logical_block_size) to legacy software.

However, this trickery soon enough led another annoying problem: because the software layer on top of the drive’s firmware is still thinking that it operates with 512B sectors internally, it could easily happen that larger logical blocks in use by the file system would become mis-aligned against physical 4K sectors that the drive is dealing with internally.

In such a case each unaligned I/O operation requested by the OS would cause the drive to perform a Read-Modify-Write (RMW) highly impacting the performance by reducing IOPS and increasing the latency. RMWs can be to a certain extent mitigated in firmware, but the drive just does not have enough information about the real needs of the OS to completely eliminate the problem.

That is why it has been finally agreed, that instead of building error-prone and extremely complicated kludges into the firmware, it makes much more sense to expose the information about the preferred sector sizes and alignment to the OS and propagate it to the upper layers such that partitioning, file system creation tools, etc. would be aware of it.

One way or another, the Linux I/O stack (starting from Linux >= 2.6.31) has been enhanced to consume vendor-provided information about the I/O limits ms-1 that allows Linux tools (parted, lvm, mkfs.*, etc.) to optimize the placement of and access to the data (see also ms-2 for other very interesting documents regarding this issue). Also, be sure to check out a very interesting article by Tejun Heo ko regarding the sector size issues in general (thanks to Slyfox for the link).

The buggy SSD firmware vs. minimum_io_size

Now, on the practical side, even though RHEL6.1, for instance, has a complete support for I/O limits, not all devices (especially the legacy ones) actually provide this information to the kernel.

Even worse, some of them, e.g. an incredible Samsung SS805 SSD drive with firmware version AD3Q (MCCOE1HG5MXP-0VBD3-0H2) which I was lucky enough to own, are broken enough to deliberately report wrong information about physical_block_size, logical_block_size and most importantly minimum_io_size to the OS (this particular drive reported 8912 bytes as the minimum_io_size).

Therefore, it is not surprising at all, that right after partitioning the disk and formating the partitions, the installation program refused to mount the newly created file systems. At the first sight, of course, the OS, and more specifically mkfs were to get the blame. How come it creates file systems that are so broken that they can not be even mounted?

A more careful investigation revealed, however, that mkfs is just doing its job: the underlying minimum_io_size hint (8K) gets propagated upwards and so it creates a file system with a 8K block size without hesitation.

However, in order for mount to be able to mount the file system, the block size should be <= kernel page size (which is 4K on x86_64 under normal conditions) lkml-1. Hence, the file system can be created, but not used.

Now that this has been figured out, all it takes is to find a firmware update to version CD3Q and magically all is well again… However, if you haven’t been previously exposed to I/O limits-related issues, would you be able to make any sense out of the mysterious EXT4-fs: bad block size 8192 messages in dmesg?

As an exercise for the readers, now forget everything that you have recollected so far and try to read the post backwards!

Updating firmware on Dell PowerEdge R710

2011-06-04T00:00:00-05:00

Introduction

Updating firmware used to be an uncomplicated, albeit slightly dangerous process. Unfortunately, as the server hardware developed over time, it did not become any simpler, but rather evolved into even more of an arcane rite.

Moreover, it seems that there is no comprehensive checklist regarding the firmware updates of the Dell PowerEdge server family and the information is spread over the support forums, tech wikis and miscellaneous blogs. Hence, this compilation was created in a hope that it will be useful and save time to some.

Let us consider a case of a Dell PowerEdge R710 featuring 2 x 2T hard drives and a 100G solid state drive, equipped with an iDRAC 6 Enterprise remote access card. There are generally several avenues that one might take to update the server firmware:

iDRAC firmware update facility (limited to the firmware of the DRAC itself and the USC, Dell Universal Server Configurator / Lifecycle Controller dell-1), which requires manually downloaded firmware update (or repair) packages
Dell USC / LC, which is an UEFI based software that is able to update almost all firmwares of the devices that are present in the server using a number of possible sources of updates:
- Dell FTP site dell-2, in which case at least one of the network interface cards present in the system has to be configured from within USC, so that the server would be able to access external resources directly or via a proxy server
- Special USB media prepared beforehand using a software called Dell Repository Manager dell-3, which needs to be deployed to a Windows management PC
- Dell OpenManage Server Update Utility dell-4, which is a DVD image containing a comprehensive collection of firmware updates; in some sense, the USB media produced by the Repository Manager are subsets of SUU specific to each particular server
Dell Systems Management Tools and Documentation all-in-one DVD dell-5, which contains the whole suite of OpenManage-branded Dell systems management software (OMSA, SBUU, SSDT and ITA)
Dell Update Packages (DUPs), which are accessible from the download pages for each specific Dell server dell-6; the right download page can be found by entering the service tag (unique server family identifier, which can be found in the DRAC interface or USC among other sources)

A rather complete (but possibly not exhaustive) list of components that might require updates is as follows:

iDRAC card
Dell BIOS
Dell USC / LC
Dell OS Drivers pack (part of LC)
Dell 32-bit Diagnostics software (part of LC)
Dell-branded RAID controllers, i.e. PERC H200I
Network interface cards
SSD devices

What follows are comments regarding each one of those and applicable update methods.

Possible firmware update procedures

iDRAC firmware update

Updating the DRAC firmware should be the first step to updating anything else, especially in the case of a restricted onsite presence / remote hands availability. During the update, the DRAC might become unavailable for a period of time up to 15 minutes; this is normal and expected.

It is preferable to turn off the server and perform the update from the DRAC management console (iDRAC Settings | Update | Upload). The update package has to be downloaded manually from the server support and drivers home page at Dell.

Additionally, iDRAC is able to re-flash USC / LC using USC repair packages lc-1 in the case if it was hosed during the update. It is generally not recommended by Dell and considered to be a last-resort action, but I have found it to be the only reliable way of updating the USC.

Dell USC / LC Platform Update

This was found to be the most reliable and complete procedure to update most of the firmwares in the system. In order to perform such an update, one needs to enter the “System Services” menu on boot (F10).

Once USC is loaded, the Platform Update can be launched. It is recommended to first update all of the suggested firmwares and then again launch Platform Update to update USC itself separately, because it was found to be the less reliable update of all.

Sometimes, the USC will get stuck after update and the server will keep displaying “Entering USC…” message for hours (everything below 20 minutes might still be fine). In this case, USC can be restored via iDRAC as described above.

Unfortunately, it took a lot of experimentation to find out that this particular server will not handle anything older or newer, but USC 1.3. There is no scientific explanation for that up to now, because generally in such a case Dell just replaces the motherboard without going on into the details of what exactly went wrong.

Dell Systems Management Tools and Documentation

Using this DVD instead of downloading the tools separately is important, because the SMTD DVD is bootable, unlike those on which the rest of the tools are distributed, where one is expected to prepare bootable disks oneself.

The process is rather trivial: one needs to boot off the DVD and launch the platform update process. The update program will request the DVD to be replaced with SUU DVD or another media containing the update repository.

It is worth to note, that SUU generally contains most outdated firmware out there, so many updates might just not be available if one goes down this route.

Dell Update Packages

DUPs are listed last, because they are expected to be run from a production operating system. Some DUPs, however, contain an ISO generator to create a bootable image which does not require an operating system to be installed, but it is not always the case.

Dell normally provides DUPs on the server support pages for Windows and Linux or upon request. There is a community-supported repository dell-linux with DUPs wrapped around with native Linux packages, however, it is not officially endorsed or supported.

Conclusion

It is worth to note, that some updates (even critical ones!) are only available in form of DUPs, i.e. for the SSD devices and not even referenced from server support pages. In such cases, one needs to search Dell support website dell-support using the model or serial number as the keyword.

These updates are not to be neglected as for instance a recent critical SSD update was to fix the minimum advertised I/O block size.

Setting up SFTP-only users on RHEL/CentOS 5

2011-05-19T00:00:00-05:00

Introduction

Normally, when a user account is created on a Unix system, the user in question can log into the system via ssh, forward TCP ports and use the SFTP subsystem if it is enabled in the server configuration. However, at some point, a sysadmin might face the need to create SFTP-only accounts for a number of users.

A naïve approach would involve creating a new user and setting his or her shell to /sbin/nologin. This solution, nevertheless, has two annoying downsides, which might ruin the security of the system, if the users are not trusted enough.

The first one is that the users would still be able to forward ports if this is globally enabled for the trusted user accounts (which includes sshd acting as a SOCKS proxy) and the second one is that the users would be able to lurk around the root file system, which certainly cannot be considered as an advantage when the users are not reliable enough.

In order to solve the first problem, one would need to apply certain configuration parameters to a specific subset of user accounts. The second issue can be avoided by instructing the SSH server to chroot into the user’s directory upon successful login.

One can conveniently apply a bunch of configuration options to specific accounts using the Match directive available in the latest versions of the OpenSSH server:

Match user badguy
    AllowTcpForwarding no
    X11Forwarding no
    ChrootDirectory /srv/sftp
    ForceCommand internal-sftp

Match group sftponly
    ...

However, this directive is only included in OpenSSH 5.x and above, whereas Red Hat Enterprise Linux 5 ships with OpenSSH 4.x. Therefore, an alternative approach will be outlined below.

In what concerns the chrooting, it can be achieved using the ChrootDirectory directive. Thankfully, corresponding patches have been backported to OpenSSH 4.x by Red Hat engineers.

Strictly speaking, it unnecessary to build a proper chroot for SFTP-only users, since OpenSSH includes a built-in SFTP implementation that does not depend upon any external libraries, but if one wants the users to be politely rejected when they try to connect via plain ssh, one could just make /sbin/nologin work and that is it.

N.B.: Red Hat Enterprise Linux 6 ships a newer OpenSSH version that fully supports the Match directive. This means that for RHEL 6 this tutorial would boil down to the configuration snippet for sshd_config presented above. Setting up a parallel running SFTP-only sshd instance is unnecessary. However, one might still wish to skim through the article for the advice on how to create a proper chroot and read the remarks in the Conclusion.

Implementation

Setting up a parallel SFTP-only `sshd` instance

Since Match directive is unavailable and the installation of extra unsupported software is to be avoided at all costs, one can bring up a parallel SFTP-only sshd instance. The proper way to go would be, of course, to create an extra RPM, i.e. openssh-sftp which installs an additional init script and symlinks, but for an one-time deployment this might be an overkill.

First, let us create an init script, which is a straightforward modification of the stock init script supplied by Red Hat:

root@box # cat > /etc/rc.d/init.d/sshd-sftponly
#!/bin/bash
#
# Init file for SFTP-only OpenSSH server daemon
#
# chkconfig: 2345 55 25
# description: SFTP-only OpenSSH server daemon
#
# processname: sshd-sftponly
# config: /etc/ssh/ssh_host_key
# config: /etc/ssh/ssh_host_key.pub
# config: /etc/ssh/ssh_random_seed
# config: /etc/ssh/sshd_config-sftponly
# pidfile: /var/run/sshd-sftponly.pid

# source function library
. /etc/rc.d/init.d/functions

RETVAL=0
prog="sshd-sftponly"

# Some functions to make the below more readable
SSHD=/usr/sbin/sshd-sftponly
PID_FILE=/var/run/sshd-sftponly.pid

# ZYV
LOCK_FILE=/var/lock/subsys/sshd-sftponly
OPTIONS=" -f /etc/ssh/sshd_config-sftponly "

runlevel=$(set -- $(runlevel); eval "echo \$$#" )

start()
{
    cp -af /etc/localtime /var/empty/sshd/etc

    echo -n $"Starting $prog: "
    $SSHD $OPTIONS && success || failure
    RETVAL=$?
    [ "$RETVAL" = 0 ] && touch $LOCK_FILE
    echo
}

stop()
{
    echo -n $"Stopping $prog: "
    if [ -n "`pidfileofproc $SSHD`" ] ; then
        killproc $SSHD
    else
        failure $"Stopping $prog"
    fi
    RETVAL=$?
    # if we are in halt or reboot runlevel kill all running sessions
    # so the TCP connections are closed cleanly
    if [ "x$runlevel" = x0 -o "x$runlevel" = x6 ] ; then
        killall $prog 2>/dev/null
    fi
    [ "$RETVAL" = 0 ] && rm -f $LOCK_FILE
    echo
}

reload()
{
    echo -n $"Reloading $prog: "
    if [ -n "`pidfileofproc $SSHD`" ] ; then
        killproc $SSHD -HUP
    else
        failure $"Reloading $prog"
    fi
    RETVAL=$?
    echo
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    restart)
        stop
        start
        ;;
    reload)
        reload
        ;;
    condrestart)
        if [ -f $LOCK_FILE ] ; then
            stop
            # avoid race
            sleep 3
            start
        fi
        ;;
    status)
        status -p $PID_FILE openssh-daemon
        RETVAL=$?
        ;;
    *)
        echo $"Usage: $0 {start|stop|restart|reload|condrestart|status}"
        RETVAL=1
esac
exit $RETVAL

CTRL+D

The configuration file would look as follows (not all directives are necessary, take into account the defaults of the distribution in use):

root@box # cat > /etc/ssh/sshd_config-sftponly
# ZYV
PasswordAuthentication no
PermitRootLogin no
PidFile /var/run/sshd-sftponly.pid
Port 2234
Protocol 2
UsePAM no

Subsystem       sftp    internal-sftp

ChrootDirectory /srv/sftp
AllowTcpForwarding no
X11Forwarding no
ForceCommand internal-sftp

CTRL+D

It is important to turn off the use of PAM for authentication in which case sshd-sftponly will fall back to reading /etc/passwd and /etc/shadow, instead of using the pluggable authentication module definitions for authentication sources and methods. Otherwise, one would need to provide an additional PAM configuration file basing upon /etc/pam.d/sshd as a template for sshd-sftponly, however, it is clearly an overkill if no advanced authentication scheme (e.g. against LDAP) is required.

Then, one needs to create a link to the sshd binary, register and start the service:

root@box # ln -s /usr/sbin/sshd /usr/sbin/sshd-sftponly
root@box # chkconfig --add sshd-sftponly
root@box # service sshd-sftponly start

Creating an SFTP-only user

The next task would be to create a group for SFTP-only users and the users themselves:

root@box # groupadd sftponly
root@box # useradd badguy -g sftponly -s /sbin/nologin -m -K UMASK=0022
root@box # passwd badguy

Here -g specifies the main group, -s sets the shell, -m creates the home directory from a skeleton and -K overrides the default options with regards to umask (optional).

It is important to set a strong password for the user (which can be immediately discarded) even though the public key authentication is to be used, because otherwise the system would consider this account to be inactive.

Now it is necessary to set up the public key authentication:

badguy@foo:~$ ssh-keygen -t rsa -b 4096

root@box # mkdir /home/badguy/.ssh
root@box # chmod 700 /home/badguy/.ssh
root@box # cat > /home/badguy/.ssh/authorized_keys
...
CTRL+D

root@box # chmod 600 /home/badguy/.ssh/authorized_keys

Creating a proper chroot

In order to avoid potential privilege escalation, the chroot and all path components leading up to the chroot have to be owned and only writable by root. Additionally, it is necessary to hardlink the shell and some supporting libraries inside the chroot:

root@box # mkdir -p /srv/sftp/{home,lib,sbin}
root@box # mkdir /srv/sftp/home/badguy

root@box # chown badguy:sftponly /srv/sftp/home/badguy

root@box # ln /lib/ld-2.5.so /srv/sftp/lib
root@box # ln /lib/ld-linux.so.2 /srv/sftp/lib
root@box # ln /lib/libc-2.5.so /srv/sftp/lib
root@box # ln /lib/libc.so.6 /srv/sftp/lib
root@box # ln /sbin/nologin /srv/sftp/sbin

Luckily, the OpenSSH server will not allow to use a chroot with wrong permissions.

Wrapping up

Now it is time to set up a new host entry and try it out:

badguy@foo:~$ cat >> ~/.ssh/config

Host box
  HostName box.com
  Compression yes
  IdentityFile ~/.ssh/id_rsa
  Port 2234
  User badguy

CTRL+D

badguy@foo:~$ sftp box

Now all these amazing feats would be for nothing if one would not tell the standard sshd daemon to deny connections for SFTP-only users and it would happily let them in:

root@box # cat >> /etc/ssh/sshd_config

# ZYV
DenyGroups sftponly

CTRL+D

root@box # service sshd restart

Conclusion

There are few additional notes, that I would like to make before closing the article.

First, if you would like to test SFTP access, you need to use sftp program as opposed to scp. It came as a surprise to me (which probably reflects one of the gaps in my knowledge due to me being an autodidact), but the widely used scp program as opposed to the popular beliefs, in fact, does not normally implement the SFTP protocol.

It is indeed the case in some operating systems, but the canonical version of scp implements the BSD RCP protocol which is tunneled through the Secure Shell (SSH) protocol to provide encryption and authentication. So bear this in mind and use sftp instead.

Another point that is worth being discussed is why one needs a separate root-owned directory tree for a chroot, instead of chrooting directly into users’ home directories. This goes back to CVE-2009-2904, when it was discovered that badass users could be able to escalate their privileges via hard links to setuid programs that use configuration files within the chroot directory. In the end it was decided that chrooting in user-owned directories actually defeats the purpose of the exercise and additional checks were introduced to restrict the possible sets of permission of the potential chroots.

Enjoy and if I have missed something in my setup please do let me know!

Transliteration matters!

2011-02-10T00:00:00-06:00

Prelude

In Russia, just as it is the case in many other states, the passport of Russian citizen is the main document that proves the identity of the holder all over the country. There are two types of passports, however. The “internal” passport is unconditionally delivered to every citizen that is more than 14 years old. The “external” or “international” passport which serves to identify the citizens of Russian Federation abroad is not, however, delivered automatically. One should additionally apply for it and as long the authorities are satisfied with the outcome of the internal checkups performed the international passport is delivered.

One small but important detail, however, is that the name of the passport holder in the internal passport is transcribed in Cyrillic script, which is a part of the writing system for the Russian language, the only national language officially recognized within the borders of Russia. Nevertheless, international treaties dictate that the name of the international passport holder should be transcribed in Latin script, which is most common among the European languages.

So, is there a way out?

Well, Russia is not the only country where a script different from Latin is employed. For instance, Chinese writing system is even more drastically different from Latin. However, in China the official set of transliteration rules exists called “pinyin” which allows one to transcribe Chinese words using Latin script as they are pronounced.

Every sound in Chinese language has an equivalent symbol in pinyin and therefore, for each word there is only one possible transformation from traditional or simplified Chinese into Latin. It is worth noting, however, that the inverse transformation is not unique for various reasons, but this detail is not important for the sake of the argument.

Obviously, a very same trick can be applied to the Cyrillic script. A lookup table can be created where each sound in the Russian language transcribed by a set of Cyrillic letters can be identified with an unique sequence of Latin letters. Even more so, such a table has already been created and adopted during the Soviet times as the “GOST transliteration algorithm” (GOST is the national Russian standards body, which basically carries countrywide the same functions as ISO, the International Standards Organization).

The theory meets practice

Surprisingly, however, Russian government could not care less about actually implementing a standard that it had created (or, rather, inherited from Soviets; could it be that they consider GOST transliteration rules to be tainted by communist propaganda or something along these lines?!).

Back in 2000, roughly ten years after the destruction of the USSR, when I had to obtain my first international passport, they were delivered by the Ministry of Internal Affairs (MID) and the transliteration was performed archaically.

That is, one could verbally express the preferences with regards to transliteration to the clerk and negotiate a compromise. If the citizen did not express any specific requests with regards to transliteration, it was to be performed by the clerk according to a vague set of rules dubbed “French” transliteration.

Later on, towards 2005, the responsibility to deliver international passports was passed on to a newly organized Federal Migration Service of the Main Office of Internal Affairs (UFMS GUVD) and the process was “streamlined”. From then on, the transliterated names were generated by the machine, however, the rules employed neither had anything to do with GOST, nor matched previously used “French” transliteration patterns.

The result, however, was claimed to be objective for which reason the citizens were denied to request any alterations. In a sense, I was lucky, because at least my last name did not change, but a very annoying extra “i” appeared in my first name (“Yuriy” instead of “Yury”).

Unfortunately, in the countries where the Latin script is in use people can not imagine that the government might change their names every once in a while, so in many cases I was refused on the grounds that the name of the recipient does not match the one in the passport. Basically, there was nothing that I could do and I just had to redo all the documents bearing my “old” name.

Where sun does not shine

Now, international passports are only valid for 5 years from the date of issue. A new law has been recently adopted which introduces so called “biometric” international passports valid for 10 years, but as usual, the implementation is lagging far behind and not only is it problematic to obtain one inside Russia, but also some consulates plainly deny such requests.

In Germany, apparently, the consulate in Berlin does handle such inquiries, but as a Russian citizen, one is not allowed to come to any Russian consulate in the foreign country of residence. One is only permitted to address to a specific consulate which is responsible for the particular area of residence, i.e. for Baden-Württenberg this would be the one in Frankfurt am Main. Of course, coincidentally, they make it very clear on their web page that they would not deliver any biometric passports yet (as of February 2011).

So, what the hell, let us go for a “normal” one and after just 3 month of waiting time it is ready to be picked up (of course, only in person)! After spending around one hour outside in the queue among angry retired immigrants I finally made it inside.

Good news is that nowadays, under the pressure from population a new service has been introduced: for a modest fee of 7.5 € you can get a stamp in your passport with your name transcribed according to the desired transliteration scheme.

Oh, wait, bummer! You can not see the passport before you request the stamp. Still, I was assuming that I need “French” transliteration, because it seemed obvious to me that they were to reissue a passport with the extra “i” I needed to get rid of. However, under the tireless lead of President Medvedev’s, apparently, the software doing the transliteration was “modernized” yet again and now, somehow, it generates names transliterated à la Française! So basically I paid for nothing…

Luckily, after some bargain the clerk agreed to cancel the stamp and make another one with my name transcribed as in the old passport, so finally I have both names on file and hopefully do not have to redo all the documents again. After all, they are all human, maybe just spoiled a little bit by the housing problem.

As a conclusion, I admit that obviously the government knows better how I would prefer to be called, but in my humble opinion just a tiny bit of consistency would definitively not hurt!

Tunneling TCP over UDP (DNS in particular)

2010-12-04T00:00:00-06:00

Introduction

This short post is just meant to be a recap on what I learned during my own very personal TCP over UDP tunneling quest.

First off, be mindful of the limitations of this technique. I have seen folks thinking that they are going to bypass TCP traffic shaping by tunneling it over UDP, since sometimes ISPs don’t implement explicit policing for UDP, at least for known protocols, such as DNS, because, frankly speaking, from the ISP’s POV it doesn’t really make much sense. More precisely, the flood of UDP packets will still reach their network, no matter whether they decide to queue and drop them to make you suffer or not.

This simply doesn’t work! UDP is mostly designed for streaming-like usage, i.e. you are not getting acknowledgements on received packets and server just goes on sending. Of course bi-directional communication is still possible in this scenario, but think of how much the performance of the applications that are designed with TCP in mind is going to degrade! It’s just not worth it. Don’t hurt your data! And if you are still thinking about re-implementing a better TCP on top of UDP (there must be no other reason why people keep polishing their TCP stacks for decades, other than that they are more stupid than you are), remember about the great demise of µTorrent <g> and think again.

Having that said, there might be valid reasons (of course, this depends on whether you include marginally unlawful activity in your definition of “valid” or not) to tunnel TCP over UDP. Such as, for instance, bypassing overly restrictive firewalls when you are set out to leak sensitive information to the outside world (an obligatory nonsensical example, since this goal can be achieved in hundreds of easier and safer ways).

Tunneling TCP streams over DNS

Now it is important to realise, that the fact that you are tunneling your traffic over UDP itself is of no help. You need to tunnel over something, that is not explicitly meant to be a bi-directional communication channel and that does not involve direct communication with the terminator of your tunnel.

For instance, there’ve been strange ideas to establish TCP over ICMP tunnels, but those are much more suitable to use as covert channels, since no sane sysop will leave such a blatant hole in his network. This is where DNS comes in. The great thing about DNS is that it is recursive by nature, which means that one can force a compliant DNS server to ask a very specific one (authoritative for the domain in question) to resolve a hostname if it doesn’t know how to do it. Also, it’s very commonly used and mostly considered harmless, which is also to our advantage.

That’s why there are so many tools for this particular purpose and they mostly work even now that many sysops are starting to recognise that security is not something that one should keep taking lightly on the networks where sensitive information is transmitted.

Overview of the available tools

All this madness started with the legendary native implementation called NSTX by Tamas Szerb, which, however, doesn’t seem to be under active development anymore and even hardly works.
Later a Java re-implementation called DNSCat by Tadek Pietraszek appeared. As the name suggests it’s more like netcat in spirit. Based upon CNAME requests, which is painfully slow, but less prone to blocking.
Dan Kaminsky (yes, the same guy which once succeeded in adding some randomness to the name resolution <g>) came up with a hacky ssh ProxyCommand compliant Perl script called OzymanDNS.
Another native effort is called iodine and seems to be pretty active, lead by two Swedish guys, Bjorn Andersson and Erik Ekman.
Yet another Java re-implementation exists by Tim Valenzuela of tcp-over-dns fame. This is the one I’ve settled with. Works with TXT records by default, hopefully the author will implement CNAME support as well.
There is an actively supported native cross-platform implementation called dnscat by Ron Bowes.

Miscellaneous hints

Here are the assorted things that I’ve learned over the past couple of days:

Absolutely make sure that your DNS set up is correct (if you try to take a shortcut and put the IP address of your end point in the NS record directly the magic won’t happen):
```
  tunnel.domain.tld.	IN	NS	ns.domain.tld.
  ns.domain.tld.	IN	A	123.123.123.123
```

If you want to map an arbitrary port (think ssh) through dnscat, do it this way:

  mknod backpipe p
  nc 127.0.0.1 22 <backpipe | java -cp ... net.ibao.dnscat.DNScatServer -o tunnel.domain.tld -p 9876 1>backpipe

The bash while syntax to make a resilient service is as follows:
```
  #!/bin/bash

  while :
  do
      /path/to/server
  done
```
If you are running Ubuntu + ufw just put your REDIRECT rule in /etc/rc.local, the rest goes to /etc/ufw/before.rules. Don’t forget to enable forwarding in /etc/ufw/sysctl.conf (reboot to apply).
The correct dig syntax is as follows (just FYI, I always keep forgetting it):
```
  dig any sub.domain.tld @ns.server.tld
```
I always forget the correct nmap syntax for host fingerprinting and keep on googling:
```
  sudo nmap -A host.tld
  sudo nmap -O host.tld
```

Nice ~/.ssh/config to use with tcp-over-dns (commented part is for DNSCat)… Enjoy a SOCKS5 proxy server on localhost:888.

  Host tunnel
    HostName localhost
    Compression yes
    ForwardX11 yes
    IdentityFile ~/.ssh/id_dsa
    #Port 22
    Port 9876
    ServerAliveInterval 30
    TCPKeepAlive yes
    User name
    #ProxyCommand $HOME/bin/dnscat-0.02/DNScatClient -o tunnel.domain.tld
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
    DynamicForward 8888

What is left to be done is probably to find a nice and easy to use socksifier for Linux, so that one, for instance, can do something along the lines of:

socksify git fetch

and enjoy happy coding while on train.

Thanks to everyone involved!

Installing Oracle Solaris 10 on Sun Blade 1000

2010-10-29T00:00:00-05:00

Introduction

So, you are a happy owner that wants to get latest and greatest Oracle Solaris running on a legendary Sun Blade 1000 workstation with glorious double UltraSPARC III CPUs and a fair 1 Gb of RAM? Follow on!

Hardware requirements

Check the version of OpenBoot firmware that your blade is equipped with. There is a known issue with OpenBoot 4.2 which prevents the workstations, having Toshiba DVD-ROM installed from booting from DVD media.

This is a highly annoying problem. In general, two kinds of advices can be found on the Internets:
- Find another DVD drive, replace the built-in Toshiba drive with a new one and hope that it will work. I have not tested this suggestion, but it might work for you.
- Find the latest firmware upgrade, re-flash the firmware and re-install the OS. This solution will be described later on this page.
Be mindful of the fact, that graphical system requires at least 768 Mbs of RAM to be available. If this is not the case, you will only be able to use the text console. In my case, I had to open up the Blade and add more RAM from a donor machine that was missing a sound card.

Re-flashing the firmware

Initially, I did not have an IDE DVD drive at hand, so I decided to go for the firmware upgrade, which proved to be more challenging that I would have ever expected. First, it took awhile to find out what is the internal Patch Number assigned to this particular firmware upgrade. For the record, it is 111292-17. Second, it turned out, that apparently you need an active support contract with Oracle to be eligible for this download. This hindrance can be effectively worked around by Googling for 111292-17.zip.

Now that the longed-for update is there, it is time to ask oneself how to proceed with the upgrade, given that the previously installed system is completely hosed (in this particular case, I have found some completely inoperable remnants of Solaris 8 scattered around the drive). The solution is easy: find a working Solaris LiveCD for SPARC, set up the networking, download the update from a private web server and proceed with the upgrade.

Investigation quickly revealed that MilaX is a perfect candidate for the job. Download the latest SPARC ISO (or Google for milax032sparc.iso if the website has already been taken down) and burn it on a CD.

Now, OpenBoot. One of the beauties of OpenBoot is that you can invoke the BIOS at any time on a running system (not that this might be the safest thing to do, though). So, in order to boot from the freshly burned image, press Stop+A on the Sun keyboard and type the following at the prompt:

ok? setenv auto-boot? false

Reboot the machine and after inserting the CD and type the following to boot from the optical drive:

ok? boot cdrom

The CD will spin up and after some time you will be presented with the login screen. Log into the system as user alex with password alex. Become root through su - with password root. Time to set the networking up and fetch the update:

# ifconfig eri0 down
# ifconfig eri0 192.168.0.1 netmask 255.255.255.0 up
# route add default 192.168.0.254
# cat > /etc/resolv.conf
    nameserver 192.168.0.254
    ...
    Ctrl+D
# wget http://server.lan/flash-update-Blade1000-latest
# wget http://server.lan/flash-update-Blade1000-old
# wget http://server.lan/unix.flash-update.SunBlade1000.sh
# chmod +x unix.flash-update.SunBlade1000.sh
# ./unix.flash-update.SunBlade1000.sh

Answer yes and keep the fingers crossed; the magic will happen and the machine will reboot itself when the update is completed.

Installing Solaris 10 proper

To be described. Refer to Blastwave. Provide some references, e.g. to Cuddletech.

Conclusion

Overall, the firmware upgrade process went smoothly and is not nearly as scary as it sounds. However, few issues, such as the difficulty to find the patch and inability to use previously installed system to perform the upgrade required to perform some prior research first.

Solaris 10 installation went extremely smoothly and left feelings of mixed sorrow and joy. Even though I did the mistake of not pre-allocating space for the ZFS database and online upgrades, I probably will not need them anytime soon, so overall it seemed to be too easy to not have a catch.

Yury V. Zaytsev

Who framed mkfs? or Winners never cheat and cheaters never win

I/O Limits: Quest for increased drive capacity

The buggy SSD firmware vs. minimum_io_size

Updating firmware on Dell PowerEdge R710

Introduction

Possible firmware update procedures

iDRAC firmware update

Dell USC / LC Platform Update

Dell Systems Management Tools and Documentation

Dell Update Packages

Conclusion

Setting up SFTP-only users on RHEL/CentOS 5

Introduction

Implementation

Setting up a parallel SFTP-only sshd instance

Creating an SFTP-only user

Creating a proper chroot

Wrapping up

Conclusion

Transliteration matters!

Prelude

So, is there a way out?

The theory meets practice

Where sun does not shine

Tunneling TCP over UDP (DNS in particular)

Introduction

Tunneling TCP streams over DNS

Overview of the available tools

Miscellaneous hints

Installing Oracle Solaris 10 on Sun Blade 1000

Introduction

Hardware requirements

Re-flashing the firmware

Installing Solaris 10 proper

Conclusion

Setting up a parallel SFTP-only `sshd` instance