Vašek's blog

Does `O_DIRECT` flag propagate over cifs/smb3 to the server?


No. At least using more-or-less default configuration at the time of writing.

Logging the usage of O_DIRECT flag

We can use the bpftrace tool to intercept open syscalls (and all its variants on the architecture we are testing). There is even an example if a similar use-case - However, it does not fit our needs as it does not say whether O_DIRECT was used. So I wrote something similar:

struct open_how {
    unsigned long long flags;

kprobe:do_sys_openat2 {
    printf("%12lld %-16s %-16s %s\n",
        (((struct open_how *)arg2)->flags & 65536 == 65536)
              ? "O_DIRECT" : "",

I won't go into detail on how exactly does the bpftrace script work. If you want to learn about scripting bpftrace, see the reference guide.

How did I end up with this exactly this script? The first important thing to note is that I am running the tests on aarch64 system. Therefore, the only open-like syscall we have is openat2, there are no other syscalls with similar meaning as they were all replaced by openat2. And because openat2 passes arguments using a pointer to a struct, we have to parse it. That's why there is a struct definition at the top of the script. Second, the maagic value 65536 corresponds to the value of the O_DIRECT flag on the aarch64 platform. To find out what the value is, we can print it out using Python:

python -c "import os; print(os.O_DIRECT)"

Output from the script might then look like this (modified for brevity):

  28805 fish          /
  28805 fish          /etc
  30373 cat           /etc/
  30373 cat           /usr/lib/arm-linux-gnueabihf/
  30373 cat           /etc/
  30373 cat           /lib/arm-linux-gnueabihf/
  30373 cat           /usr/lib/locale/locale-archive
  30373 cat           /etc/passwd
  28805 fish O_DIRECT /home/vasek/.local/share/fish/fish_history

Opening file with O_DIRECT for test

To open a file with O_DIRECT flag, we can use something like this:

dd if=/dev/zero count=1 bs=16 of=outfile oflag=direct

When we run in, the bpftrace script prints out this:

  675 dd  O_DIRECT  outfile

The actual test

With the tools we prepared, testing whether O_DIRECT propagates over the network filesystems is simple. I tried mounting the disk using all of these commands:

sudo mount -t cifs -o user=vasek //server/vasek /mnt/vasek
sudo mount -t smb3 -o user=vasek //server/vasek /mnt/vasek
sudo mount -t cifs -o user=vasek,cache=none //server/vasek /mnt/vasek
sudo mount -t smb3 -o user=vasek,cache=none //server/vasek /mnt/vasek

And with all of these, the bpftrace script printed out something like this:

  24447 smbd           outfile

So, we haven't seen the O_DIRECT flag propagated over the network.

Server config

And just for completeness, the config used for smbd was the following. Everything else was left as a default based on the distro's package.

path = /mnt/biggadisk/vasek
create mask=0777
directory mask=0777
Thoughts? Leave a comment