[ Home | Weather | Wiki | HN | News | xkcd ] [ Search | Settings | About ] [ Light | Dark ]
Jails on FreeBSD
[ Top | New | Ask | Show | Same poster | Same domain | Source site ]
Posted on Thursday, May 25th 2023 by rodrigo975https://ogris.de/howtos/freebsd-jails.html
[ Threaded | Oldest | Newest ]
@ Friday, May 26th 2023 by jmclnxVery nice
I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.
@ Friday, May 26th 2023 by betaby | ParentPlease provide technical details of `Jails are far more secure`.
@ Friday, May 26th 2023 by throw0101c | ParentMultiple CVEs for escaping? E.g.:
The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs:
@ Friday, May 26th 2023 by betaby | ParentNo? Because many of them Docker specific. Neither I mentioned Docker anywhere, not Docker is a good example of software with a good security record. Another link lumps together k8s, runc and so on. I specifically mentioned systemd-nspaw in the beginning as contender to jails, not other solutions.
@ Friday, May 26th 2023 by tptacek | ParentMisleading: the most important vulnerabilities that break shared-kernel isolation systems like Jails, Zones, and containers are kernel LPEs, and they aren't reported as "escapes".
@ Friday, May 26th 2023 by gtirloni | ParentWhat makes Jails so far ahead of containers in your opinion?
@ Friday, May 26th 2023 by doublerabbit | ParentNot op, but for me:
You can setup Linux in a jail.
You can nest bhyve virtual machines in a jail.
You can assign individual NICs to a jail.
You can encrypt with ZFS a jail.
You can run browsers in a jail with full set of features.
@ Friday, May 26th 2023 by betaby | ParentAll of those available on Linux and even more, see https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.h...
@ Friday, May 26th 2023 by doublerabbit | ParentI don't see the guide to run FreeBSD in a container from that man page.
And what about if my Linux Distro doesn't use system-md, like Slackware.
@ Friday, May 26th 2023 by betaby | Parent>And what about if my Linux Distro doesn't use system-md, like Slackware.
It's a choice of Slackware not use systemd infra. However on Linuxes you have more that one choice to run containers, your next option would be LXC/LDX https://docs.slackware.com/howtos:misc:lxc
@ Friday, May 26th 2023 by doublerabbit | ParentSure and my choice not to use it too. Well aware of LXC too.
I'm not claiming Linux can't. I was stating why. I know Linux can do what FreeBSD can do, but BSD does it better.
However both are pretty much mimicked concept from Solaris Zones
@ Friday, May 26th 2023 by sillystuff | Parent>However both are pretty much mimicked concept from Solaris Zones
Your timeline is off.
FreeBSD and Linux were within a year of each other (not in-tree on linux, but Debian packaged VServer kernels and VServer + GRSec kernels; we used vserver+grsec debian packaged kernels at work in the mid through late '00s).
Solaris containers came years later.
2000 - FreeBSD jails
2001 - Linux VServer
2004 - Solaris Containers
@ Friday, May 26th 2023 by 5e92cb50239222b | ParentI wonder why would anyone spend time on developing and maintaining that. FreeBSD needs Linux userland and syscall support to get roughly the same level of software compatibility. The opposite isn't true (maybe it is for your special snowlake internal application developed specifically for FreeBSD, but it isn't so in general case).
All other things you have mentioned are possible with pretty much any of the Linux container runtimes. systemd-nspawn is just one of them, and I don't think it is actually used very much compared to the alternatives.
@ Friday, May 26th 2023 by jmillikin | ParentMost of those are supported by "containers", which are after all just a chroot and a process tree with restricted visibility into certain kernel objects. Assigning individual NICs or running a browser with extra isolation is one of the classic Linux namespace tutorials.> You can setup Linux in a jail.I don't see how this can be possible. Could you explain more how to boot a Linux kernel within a FreeBSD jail?
edit: I'm not talking about running binaries compiled for Linux under FreeBSD. The parent said it's possible to set up Linux within a jail, so I want to see instructions to boot an actual Linux kernel as a FreeBSD process.
@ Friday, May 26th 2023 by rwaksmunski | ParentYou use FreeBSD kernel and Linux userland. FreeBSD can emulate Linux system calls.
@ Saturday, May 27th 2023 by umanwizard | ParentThere is no such thing as "Linux userland". Obviously you can port most GNU utilities and other things that are typically run on Linux on FreeBSD but that's a far cry from "running Linux".
@ Friday, May 26th 2023 by NexRebular | ParentUsing the linuxulator.
@ Friday, May 26th 2023 by throw0101c | Parent>Assigning individual NICs or running a browser with extra isolation is one of the classic Linux namespace tutorials.
With FreeBSD you can set up a standalone virtualized network stack:
@ Friday, May 26th 2023 by the8472 | ParentI read the quote you're replying to as "you can also do this under linux", i.e. it didn't dispute that freebsd jails can do it. But it's not a distinguishing feature of jails.
@ Saturday, May 27th 2023 by chungy | ParentYou could maybe build and run User-Mode Linux on FreeBSD, and run that in a jail. That's pretty close to your own interpretation of "Linux kernel in a jail"
But to be real: that's obviously not what was meant. If someone wants to install and run Ubuntu inside a FreeBSD jail, it is well possible to do that. Nobody really cares if it's running kernel.org code, just that the binaries are running as expected.
@ Friday, May 26th 2023 by irusensei | ParentI don't know about the low level stuff but LXC is pretty much the same end result as FreeBSD jails. At least for my very basic needs that is.
@ Friday, May 26th 2023 by johnisgood | ParentWhat about `firejail` on Linux? Is it not much more secure than FreeBSD Jails which is just based on chroot?
@ Friday, May 26th 2023 by bell-cot | ParentFreeBSD Jails are conceptually just based on chroot. Their actual design and implementation are a few orders of magnitude more secure.
@ Friday, May 26th 2023 by mardifoufs | ParentA few order of magnitudes? Can you be more specific? I'm genuinely wondering what you could be referring to that I've missed.
@ Friday, May 26th 2023 by tptacek | Parentchroot more or less isn't secure at all. All they're really saying is that Jails aren't built on top of chroot, but rather are a ground-up reimplementation motivated by improving chroot.
@ Friday, May 26th 2023 by mardifoufs | ParentAh! For some reason I thought jails were built on something similar to chroot at the kernel level, meaning that it also came with the same security caveats.
(to be more accurate, I knew that it was not "on top" in the same way as Linux containers are built on top of chroot. )
@ Friday, May 26th 2023 by bell-cot | Parenthttps://en.wikipedia.org/wiki/Operating_system-level_virtual...
@ Friday, May 26th 2023 by NexRebular | ParentAlso the SANE paper:
@ Friday, May 26th 2023 by johnisgood | ParentThanks, I'll check out the specifics.
@ Friday, May 26th 2023 by tptacek | ParentJails are not, in fact, far more secure than Linux containers. Arguably they're a better, more coherent design, but both are shared-kernel isolation schemes, and the high-order bit of their security is exactly the same. The best argument you'll come up with is that FreeBSD kernel LPEs are rarer than those of Linux, but now we're just back to the standard Linux vs. FreeBSD security debate, which never goes anywhere.
@ Friday, May 26th 2023 by zie | ParentI agree with all of this, with perhaps one tiny addendum. Defaults matter, and last I checked Docker/container defaults are miserable for security. I'm guessing if you randomly inspected production container vs jails out in the world, way more container solutions would have giant holes in them compared to jails.
@ Friday, May 26th 2023 by cpuguy83 | ParentDefaults on Docker specifically are pretty locked down.
Even if if the container process has the root uid it is not able to do most things normal root can do, and is pretty much limited to being able to access root owned files and binding to privileged ports (which in a container doesn't tend to matter).
@ Friday, May 26th 2023 by yjftsjthsd-h | Parent>Defaults on Docker specifically are pretty locked down.
That is strongly dependent on your threat model. The default docker configuration completely bypasses the firewall, making it trivial for containers to be exposed to the open internet with no way for admins to prevent it. Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.
 It is quite easy for someone even slightly inexperienced to accidentally write, say `-p 1234:1234` instead of `-p 127.0.0.1:1234:1234` and thereby cause a security incident or near-miss; ask me how I know.
@ Friday, May 26th 2023 by cpuguy83 | ParentDocker has to run as root, or use otherwise insecure methods ("rootless" is a sham, it requires suid binaries and CVE ridden unprivileged user namespaces).
I agree with ports, working on it.
@ Friday, May 26th 2023 by cpuguy83 | ParentThat said, docker does have support for running "rootless"
@ Friday, May 26th 2023 by ipdashc | Parent>Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host.
I never got why this is commonly used as an argument against Docker, TBH. You just don't give out access to the Docker socket to anything untrusted. Doesn't pretty much everyone know that by now?
I feel like people always say Docker is awfully insecure, but then the proofs-of-concept include flags like `--privileged`, or the socket is mounted, or / is mounted, or --net=host is set... etc. Docker by default always seemed pretty good to me, but I'm not very experienced in that realm, so I'm just wondering what I'm missing.
@ Friday, May 26th 2023 by 1over137Instruction are for FreeBSD 12, I wonder if it's still current for 13 and 14...?
@ Friday, May 26th 2023 by BanazirGalbasi | ParentIn my limited experience with BSD over the years (I tinker with it every year or so), a lot of the core experience remains the same. Additions to rc.conf should keep working, as well as configuration files for things like jails.
@ Friday, May 26th 2023 by UI_at_80x24 | ParentNot really.
Specifically the suggestion to use lo# instead of vnet.
@ Friday, May 26th 2023 by st3fanI upvoted this because I have a lot of love for FreeBSD and I think Jails are great.
However, I really wish the FreeBSD folks would educate themselves a bit more about what is actually available on Linux and how those options compare to FreeBSD Jails.
@ Friday, May 26th 2023 by doublerabbit | ParentAnd vice versa. I really wish the Linux folk would educate themselves a bit more about what is actually available on FreeBSD.
@ Friday, May 26th 2023 by tptacek | ParentWhy would they? What difference would it make to them?
@ Friday, May 26th 2023 by adr1an | ParentWho are you referring to? Linux or FreeBsd people? I guess both should look to each others'. But it's common to see this in FLOSS, right? Goes beyond namespaces/ isolation.
@ Friday, May 26th 2023 by tptacek | ParentI don't mean to sound snide, I'm just saying: if your stack is Linux and the container ecosystem (for instance, if your stack is K8s) what would be in it for you to investigate Jails?
@ Friday, May 26th 2023 by mnd999 | ParentIf your stack is K8s and you actually need K8s then none. Most people using K8s don't actually need it and are practicing resume driven development.
@ Friday, May 26th 2023 by tptacek | ParentYou get that resume-driven development is a real thing, right? It's not irrational. Improving your future prospects is a very legitimate reason to invest in K8s development.
I'm speaking against my own interests here; I don't like K8s.
@ Friday, May 26th 2023 by bojo | ParentI think the point was aimed at people that want to debate about jails vs. linux namespaces. Not much of a debate if you don't understand the strengths and weaknesses of both sides.
If they're invested in an existing linux container ecosystem and don't intend to change then yeah probably not much gained.
@ Saturday, May 27th 2023 by arp242 | ParentThe biggest advantage is an abstract one; it's usually a good idea to have some idea of different ways to solve a problem even if you're going to be mostly (or even exclusively) focused on one particular way. "Narrowness of experience leads to narrowness of imagination" and that kind of stuff.
For example years ago I started runit in a container to run multiple processes (multiple containers was a bit tricky for operational reasons) and some people were surprised: "oh wait, you can do that?" Who knows if my experience with jails contributed to this "idea" (if you can even call it that), but it probably didn't hurt.
@ Friday, May 26th 2023 by mnd999 | ParentWell, they might realise their use case it's better suited to FreeBSD. But probably not better-enough to switch OS.
But mainly they might make make more educated comments. Sometimes, and maybe this is more a historic thing, the Linux advocates can get a bit carried away.
@ Friday, May 26th 2023 by thehigherlifeSince this is gaining some traction. I'm curious if you are using jails and a jail manager, what are you're using now? I'm still using IOCAGE (Version 1.2) on freeBSD 13, but it seems like its not the best way to do it anymore? Documentation is all over the place on what is the best way to manage jails now.
@ Friday, May 26th 2023 by UI_at_80x24 | ParentI'm in the same boat.
Started with ezjail, switched to iocage, now thinking about bastille or roll-my-own.
@ Friday, May 26th 2023 by kitsunesoba | ParentStill on ezjail myself. Set my NAS/home server up several years ago with a blog post tutorial centered around ezjail and have just carried that forward. Eventually will need to do a fresh reinstall though and would be interested to know what's considered best practice these days.
@ Friday, May 26th 2023 by kevans91 | Parentiocage as well... I've been trying to sucker someone into writing a new jail manager in flua, since we have that in base. It hasn't worked yet.
@ Friday, May 26th 2023 by soupbowl | ParentI use BastilleBSD, it has a simple template system which I really like and decent docs.
Bastille also has a sister project 'rocinante' which allows you to use Bastille templates on
the host. I converted my ansible scripts to bastille templates and it works a lot better for *ME*. I found I spend more time updating ansible scripts whenever I needed to use them, it costed more time then just using a setup.sh script, which rocianate basically is.
Another new kid on the block for jails is AppJail, it has some interesting features. I have not played with it enough to say how stable it is.
@ Friday, May 26th 2023 by soupbowlThere are a lot of jail tools these days, it would be nice if we posted more modern FreeBSD guides.
Myself I use BastilleBSD heavily for personal and small business use.
@ Friday, May 26th 2023 by codetrotterLet me chime in and say, Michael W Lucas has an awesome book about FreeBSD Jails. I bought it recently and I read the whole thing and it helped me a lot. I still had to figure some things out by myself because the book is for a slightly older version of FreeBSD. But it is an awesome book.
I use only the tools included in base system for setting up my jails. No "ezjail" or anything.
If you read his whole book you will see how it might be the correct choice to just do it yourself. Depending on what you want to do etc.
For me I am definitely much better off having set it up myself with the help of mwl's book.
Buy the physical copy of the book.
PS: Use vnet interfaces for most of your jails.
@ Friday, May 26th 2023 by chunk_waffle | ParentAll of Michael W Lucas' books are great, they're so easy to digest.
@ Friday, May 26th 2023 by mikeceIs there a "best article" out there comparing BSD Jails to Docker and the strengths and weaknesses of choosing one or the other?
(I'm also curious if BSD Jails are the same thing as Solaris Zones but with a different name or if there is significant nuance making them different).
@ Friday, May 26th 2023 by tptacek | ParentUse whichever one you're more comfortable with and that suits the rest of your stack best. There's nothing Jails are going to allow you to do that Linux containers can't do. All the shared-kernel workload isolation schemes are comparably safe.
@ Friday, May 26th 2023 by whalesalad | Parenthttps://blog.jessfraz.com/post/containers-zones-jails-vms/
@ Friday, May 26th 2023 by paulddraper | ParentFor starters, Docker runs as a daemon.
@ Friday, May 26th 2023 by __turbobrew__ | ParentFor starters starters, Docker is a few layers up in the stack from jails. You would be better off comparing runc to jails or just cgroups+namespaces to jails.
@ Friday, May 26th 2023 by paulddraper | ParentNo.
Docker and runc are very similar.
Docker run Docker containers. runc runs OCI containers.
And jails operate at a similar level, though without an image format.
@ Friday, May 26th 2023 by __turbobrew__ | ParentRight, so jails do not have images or any of the tooling around building and distributing images which is the vast majority of what makes up docker. This is why cgroups+namespaces are much closer to jails than docker is.
There are systems which build upon jails such as iocage and ezjail which are more similar to docker.
@ Friday, May 26th 2023 by jmillikinWhenever an article about FreeBSD's jails gets posted here, the comments tend to drift toward an argument about jails vs Docker containers. People call them "Linux containers", but really they're talking about Docker (or its clones like Podman).
This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error.
Questions I would like to see answered in that article:
* Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
* Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
* Can live processes be moved into or out of a jail?
* Can jails be used to make a process think it's running as a different user?
I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.
@ Friday, May 26th 2023 by throw0101c | Parent>Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack:
You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host:
But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
>Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
>Can live processes be moved into or out of a jail?
No: jails are VM-like in functionality.
>Can jails be used to make a process think it's running as a different user?
The jail, being VM-like, would have its own passwd.
@ Friday, May 26th 2023 by the8472 | ParentCalling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like.
The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.
Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
@ Friday, May 26th 2023 by 0x457 | ParentIt's called VM-like because:
- Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1)
- Normally has its own network stack (VIMAGE)
- Normally runs a bunch of background services like a regular VM or on-metal would.
- Normally has the entire copy of user-land
- I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM.
The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux).
To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
@ Friday, May 26th 2023 by the8472 | ParentThat's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks.
The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
@ Friday, May 26th 2023 by fragmede | Parent>That's just saying how people tend to use it
People tend to use words that describe how they use things to describe what they are.
@ Friday, May 26th 2023 by righttoolforjob | ParentWell, take a car as an example.
The question is not what you use a car for, but rather how the car is built.
@ Saturday, May 27th 2023 by 0x457 | ParentCar is a carriage-like mode of transportation.
The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything."
I've provided examples why "VM-like" is used to describe jails.
@ Friday, May 26th 2023 by 0x457 | ParentI think you need to look into usage of "-like" suffix.
@ Friday, May 26th 2023 by 0x457 | Parent>But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.
@ Friday, May 26th 2023 by toast0 | Parent>But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.
It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.
@ Friday, May 26th 2023 by tptacek | ParentYou can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs.
If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).
@ Friday, May 26th 2023 by toast0 | ParentWell what are you going to do when you have to run something you don't trust?
Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust.
Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage.
Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.
@ Friday, May 26th 2023 by tptacek | ParentIf you really don't trust it, you run it inside a virtual machine.
The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf.
A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.
@ Friday, May 26th 2023 by sophacles | ParentWorth pointing out that VM escape is a thing too: https://en.wikipedia.org/wiki/Virtual_machine_escape
@ Friday, May 26th 2023 by tptacek | ParentIt is. It's a much smaller attack surface.
@ Saturday, May 27th 2023 by GauntletWizard | ParentI'm unconvinced it is. All of those virtual hardware drivers require their own code, often at kernel privilege. Hypervisor functionality at the processor level helps but is a huge attack surface in itself. X86 "rings" are under tested because it's hard to - many violations just kill your whole VM, but more importantly there's a ton of instructions that access "physical" memory and the state of the page mapper, even at the ring-0 level, matters a lot.
I do think VMs are likely more secure than containers in a cloud environment, because of course at that level you have both problems, but I don't believe that.the number of vulnerabilities found at the VM layer is at all reflective of their actual vulnerability.
@ Saturday, May 27th 2023 by tptacek | ParentIt simply is. Even in a heavyweight VM with lots of hardware support, those hardware drivers are a tiny fraction of the user/kernel interface, no matter how you choose to count it (lines of code, number of foreign calls, number of exposed modules).
If you don't want to derive this axiomatically, fair enough: count vulnerabilities. The tally you're looking for is every Linux LPE versus every Linux KVM escape.
@ Friday, May 26th 2023 by heavyset_go | Parent>I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /
When you do this, do the jails/chroots act like separate overlays on top of / to the daemons?
For example, if you use systemd-nspawn or systemd-run, you can run daemons in containers that get their own view of /, but writes take place in separate overlay file systems.
I'm curious how that compares with what you're doing with jails.
@ Friday, May 26th 2023 by toast0 | ParentNo, my jailed deamons don't have an overlay, writes manifest to the filesystem the same as if they weren't jailed (which is good for me). Someone elsewhere described a vps they user which did have an overlay system, but I haven't used that.
@ Friday, May 26th 2023 by tptacek | ParentJails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.
@ Friday, May 26th 2023 by heavyset_go | ParentI believe I recall there being a period quite a while ago when things like LXC and OpenVZ were described/marketed as being "VM-like" around the same time that jails were popularized.
It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.
@ Friday, May 26th 2023 by tptacek | ParentI don't know about any of that. LXC containers are also not VM-like.
@ Friday, May 26th 2023 by fragmede | Parent>The defining attribute of a VM is that it runs its own kernel.
of a VM.
The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck.
Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place.
Or to put it another way, strcmp("VM", "VM-like") != 0.
@ Saturday, May 27th 2023 by insanitybit | ParentIf VM "like" means "doesn't do the one thing that VMs do aka run a separate kernel" it's not a very useful term
@ Saturday, May 27th 2023 by bear8642 | Parent>>Can live processes be moved into or out of a jail?
>No: jails are VM-like in functionality.
Could this be implemented though if moving between sufficiently similar operating systems?
Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.
@ Saturday, May 27th 2023 by inkyoto | Parent>Jails are basically like a VM-like/light [...]
With all due respect, they are not.
The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).
Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.
VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.
@ Friday, May 26th 2023 by toast0 | Parent>* Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?
Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation.
>* Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?
I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this?
>* Can live processes be moved into or out of a jail?
A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails.
>* Can jails be used to make a process think it's running as a different user?
I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?
@ Friday, May 26th 2023 by heavyset_go | ParentHate to nitpick, but a comparison to bare namespaces wouldn't be apt, either, as Docker et al use more features than just namespaces in their container implementations, like control groups, union/overlay/snapshot file systems, nftables, etc.
For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.
@ Friday, May 26th 2023 by assimpleaspossi | ParentThe article is about how to set up jails on FreeBSD.
The article has nothing to do with anything you are talking or asking about.
@ Friday, May 26th 2023 by vermaden | ParentFirst of all - the Namespaces and Cgroups on Linux are not about security. They are about features.
FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed.
To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure.
@ Friday, May 26th 2023 by icedchai | ParentYou're telling those namespace / cgroup features have nothing to do with security? Not in any way, shape, or form? I'm a little skeptical.
@ Saturday, May 27th 2023 by jmillikin | ParentNamespaces and cgroups provide resource accounting and some limited isolation between trusted workloads. They're not considered a sandbox or security boundary because the processes have full access to the Linux kernel APIs, which are not well-hardened.
To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.
@ Saturday, May 27th 2023 by icedchai | ParentI agree containers are more for ease of deployment, not security. But security is about shades of gray... limited isolation is one of those.
@ Saturday, May 27th 2023 by user6723 | ParentUnderrated post.
@ Friday, May 26th 2023 by craftkiller | Parent>Can live processes be moved into or out of a jail?
That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) , and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa.
@ Friday, May 26th 2023 by alberthOff topic: I miss the days when web pages looked like this link.
High info density, consistent look, "responsive" without being responsive.
@ Friday, May 26th 2023 by layer8In the aughts (2000s) I rented a "managed" VPS that was based on FreeBSD jails and an overlay filesystem. It was great in that you didn't have to maintain most of the software yourself (e.g. security updates) because it was maintained by the hoster on the base filesystem layer, while at the same time you were still root on your own filesystem overlay, where you could add or modify any file from the base filesystem. If you messed up somehow, you could simply revert to the base filesystem version of the respective files. Furthermore only the files on your overlay counted against your filesystem quota. Unfortunately the hoster discontinued that offering after a couple of years.
@ Friday, May 26th 2023 by porker | ParentVerio VPS FTW! That was the first proper hosting I had.
@ Friday, May 26th 2023 by codetrotter | Parent>Unfortunately the hoster discontinued that offering after a couple of years.
I'm planning on starting a FreeBSD based hosting service, but I am not sure if anyone actually wants what I have in mind.
The thing is that I have some couple of very specific ideas in mind.
I will offer a small amount of storage, and not general internet connectivity.
Users will have 25GB space and the idea is that they will be able to connect over Wireguard, but they cannot make outbound connections from the host.
It'll be like a /home/user in the cloud.
So you can keep some files there and connect from wherever in the world and use the cli tools we all know and love. But idk if there is an actual market for that or not.
The selling point is that I will be focusing on the storage of that data. ZFS with redundancy and offsite backups - the works. That's why the amount of data should be low. Only keep the most important files there.
Also I'm gonna accept payments in Bitcoin only, and people will have to sign up for many years upfront.
I think maybe the number of people that want this can be counted on one hand heh
@ Friday, May 26th 2023 by gtirloni | ParentOne question you may want to answer is why anyone would use your solution vs something like rsync.net or a small VM in DigitalOcean, etc. You said the selling point would be storage. What would be the differentiator in that area? Would price be it?
Just trying to help with question to validate the idea. Best of luck.
@ Friday, May 26th 2023 by codetrotter | Parent>something like rsync.net
My impression of rsync.net is that it is for backup. Whereas my service would be a live service you ssh into. Basically, connect to Wireguard VPN and then ssh into your cloud home.
The idea being that this is where you keep, and interact with, your files that are important.
My service would have less system administration for the user, and high level of storage redundancy and offsite backups
>Just trying to help with question to validate the idea. Best of luck.
Thank you, I appreciate it :)
@ Friday, May 26th 2023 by gtirloni | ParentGot it! This: less system administration, that's a good selling point!
@ Friday, May 26th 2023 by layer8 | ParentNo outbound connections would likely be a deal-breaker for many.
@ Friday, May 26th 2023 by jedbergJust one nit, FreeBSD had jails before March of 2000. I know this because I was using them in 1999. And in fact they were merged into CURRENT in early 1999 and STABLE in mid/late 1999. It was merged into RELEASE in March 2000.
@ Friday, May 26th 2023 by erk__ | ParentThey were actually developed a year before, but the person who paid for development had a year of exclusive use.
@ Friday, May 26th 2023 by gumballindieEvery time I think of freebsd and how cool it was and is I wondering what caused linux to overtake it. I like linux as well, but I just don't understand the mechanics behind freebsd not gaining traction.
@ Friday, May 26th 2023 by jbverschoor | ParentFor me it was Java. I think there was something with mmap().
I ran FreeBSD servers with jails around '00. They worked fine.
Not for me, but in hindsight, maybe the FreeBSD daemon mascotte had something to do with it.
But around that time bea weblogic and ibm websphere also became popular, and I don't think Java was officially supported. It was also a period where people tended to run oracle, but I'm not sure if that tab on Linux or something else. I think it was some custom Linux "unbreakable oracle"
@ Saturday, May 27th 2023 by chungy | ParentAT&T v BSDi had a large hand in that outcome of fates. It put the legality of BSD code in all its places into doubt for many years.
Ironically, just a few years later, SCO thought "hey that was a good idea" and did it to Linux too.
@ Friday, May 26th 2023 by DetrytusOfftopic, but isn't it funny that the system that calls itself FreeBSD has a concept of "jails"? Couldn't they come up with better name? :)
Search Hacker News
Hacker News provided by Y Combinator and Algolia.
These pages best viewed with Netscape Navigator 1.1 or later.