Linux / systemd

systemd EnvironmentFile Keeps Your Inline Comments

Your service will not start. The setting it is complaining about looks correct in the file. You read the line, you read it again, and it is fine. On a unit using EnvironmentFile=, the comment you wrote after the value is now part of the value.

The symptom

The unit restarts over and over. Check it with command systemctl show yourapp -p ActiveState,SubState,Result and you get activating, auto-restart, exit-code.

Then read the application's own log. The error names a value you can see is correct:

Config error: LOG_LEVEL must be one of DEBUG|INFO|WARN|ERROR,
got "INFO                 # DEBUG | INFO | WARN | ERROR"

The comment came along with it.

Why it happens

systemd only treats # as a comment when it is the first thing on the line. An end-of-line comment is not a comment to systemd. It is more value.

So this line:

LOG_LEVEL=INFO                 # DEBUG | INFO | WARN | ERROR

Sets LOG_LEVEL to the whole string, spaces and all.

This surprises people because most .env parsers do strip inline comments. Python dotenv strips them. Node dotenv strips them. Docker Compose env_file strips them. systemd does not.

BE WARNED: your app's own parser cannot save you here. Most .env libraries refuse to overwrite a variable that is already set in the environment. systemd set it before your process started. So the library never gets a turn, and the same file that parses perfectly when you run the app by hand fails under the unit.

Confirm it in ten seconds

Ask systemd what it is actually going to inject:

systemctl show yourapp -p EnvironmentFiles
systemctl show yourapp -p Environment

If Environment shows your values with the comments still attached, that is the whole bug.

Check whether your unit even uses the file with command grep -E '^\s*EnvironmentFile' /etc/systemd/system/yourapp.service.

Fix one: take the comments off the value lines

Move every comment to its own line.

# DEBUG | INFO | WARN | ERROR
LOG_LEVEL=INFO

Restart and it works. This keeps EnvironmentFile= and keeps your documentation.

It also breaks again the first time somebody tidies the file back up.

Fix two: stop letting systemd parse it at all

Better if your app already loads its own .env. Delete the EnvironmentFile= line and let the application's parser do the job, since that one handles inline comments correctly.

[Service]
WorkingDirectory=/opt/yourapp
# No EnvironmentFile= on purpose. systemd keeps inline comments as part of the
# value. The app's own loader strips them, and it will not override a variable
# systemd already set, so systemd must not set them at all.
ExecStart=/usr/bin/node dist/app.js

Keep WorkingDirectory. That is what lets the loader find ./.env.

Then run systemctl daemon-reload. A unit already sitting in a restart loop will pick up the corrected file on its next attempt and start on its own.

The dangerous version of this bug

The crash loop is the lucky outcome. It only happens because something validated the value and rejected it.

Numbers and booleans usually fail loudly. A number parser gets NaN and throws. A boolean parser gets a string that is not true or false and throws.

Plain strings do not. A URL, a path, a hostname or a token with a comment glued on the end is accepted silently and used wrong. You get a connection failure that points nowhere near the real cause.

BE WARNED: check your string settings first, not your numbers. The numeric ones announce themselves. A BASE_URL carrying half a sentence of documentation will just look like the remote end is broken.

If a deploy script writes your unit file

Fix the template, not the installed file.

Plenty of install scripts do install -m 644 myapp.service /etc/systemd/system/ and then sed a path into it. Edit the running unit and your next deploy overwrites the fix and puts the bug straight back.

Confirm which file is the source with command grep -n 'service' deploy.sh and edit that one.

How to spot a unit that has never actually run

This bug can sit in a repo for months, because it only fires when systemd starts the thing.

Compare the timestamps:

stat -c '%n %y' /etc/systemd/system/yourapp.service
systemctl show yourapp -p ExecMainStartTimestamp

If the unit file is newer than the last successful start, every run you remember was you launching it by hand. Your unit is untested. Start it once, on purpose, before you rely on it.

When this isn't your problem

Your unit uses Environment= instead. Those are quoted in the unit file itself and behave differently.
You use Docker Compose env_file. Compose strips inline comments.
Your values are quoted. systemd honours quoting, so LOG_LEVEL="INFO" # note behaves. Do not rely on remembering to quote every line.

The general lesson

Two different parsers reading one file will eventually disagree, and the disagreement shows up as a bug in something else entirely. Pick one parser. If systemd and your application both read the same .env, decide which one owns it and take the other one out of the path.