Tracking Files with Git: .gitignore, git rm --cached and git mv

Version note: The commands in this post were tested with Git 2.54.

Guess First

Last week, before the repo had a .gitignore, you accidentally committed a .env file with an API key in it. Today you noticed, added a .env line to .gitignore and changed the key in the file. What does git status say: is .env invisible now, or is it still listed as "modified"?

In this post we look at what .gitignore does and, more importantly, what it does not do. Along the way we add the other file-tracking tools: git check-ignore, git rm --cached, git mv and .gitattributes.

The Problem: Things That Don't Belong in the Repo

First, let's actually make the mistake from the opening. We had added a license file and a .env with local settings to the bookstore repo:

# Last week, before there was a .gitignore
$ git add LICENSE.txt .env
$ git commit -m "Add license and local settings"
[main 5517793] Add license and local settings
 2 files changed, 4 insertions(+)
 create mode 100644 .env
 create mode 100644 LICENSE.txt
$ git log --oneline -3
5517793 (HEAD -> main) Add license and local settings
e3ef418 Add cart total
fae4454 Add shopping cart

Then the project grew: dependencies were installed, a build was made, the app ran and wrote a log. macOS left a .DS_Store, as it does in every folder. We also created .env.example as a sample settings file for the team:

# After npm install, a build and a run of the app
$ git status -s
?? .DS_Store
?? .env.example
?? debug.log
?? dist/
?? node_modules/

The only file in this list that belongs in the repo is .env.example. The node_modules/ folder contains thousands of files and can always be reinstalled from the file that lists the dependencies (package.json in Node projects). dist/ is the build output, always reproducible from the source. debug.log is specific to your machine. .DS_Store is an operating system file that has nothing to do with the project. This noise makes git status unreadable, and a single git add . puts all of it into the repo.

The First .gitignore

We create a file called .gitignore in the root of the repo and write this:

# Dependencies
node_modules/

# Build output
dist/

# Logs
*.log

# Local settings
.env
.env.*
!.env.example

$ git status -s
?? .DS_Store
?? .env.example
?? .gitignore
$ git status -s --ignored
?? .DS_Store
?? .env.example
?? .gitignore
!! debug.log
!! dist/
!! node_modules/

The noise is gone. Of the three remaining files, .gitignore and .env.example are waiting to be committed; we will deal with .DS_Store in a different way shortly. The --ignored option also shows ignored files, marked with !!; it is handy for seeing what is really being ignored.

.gitignore is an ordinary file, and it gets committed to the repo. That way the rules apply to the whole team. Think of the "no bags or food" sign at a library entrance: the rule is posted once, and everyone runs into the same rule.

Patterns and Exceptions with !

The few lines we wrote use almost the whole pattern language. Lines starting with # are comments, and blank lines are ignored. * matches anything except a slash: *.log catches log files in every folder. node_modules/ and dist/, with a trailing slash, only match directories. A leading slash anchors the pattern to the folder the .gitignore is in: /build only catches build at the root and leaves src/build alone. ** crosses folder boundaries: **/logs catches logs at any depth, logs/** catches everything under that folder.

A line starting with ! is an exception: it brings back a file that an earlier rule caught. .env.* ignores every variant such as .env.local or .env.production. But that rule would also catch .env.example, and !.env.example rescues it. Order matters: Git reads the lines from top to bottom, and the last matching rule wins.

There is one limitation: you cannot bring back a file inside an ignored folder with !. After dist/, a !dist/keep.txt line does nothing, because Git never looks inside an ignored folder. You have to ignore the folder's contents rather than the folder itself: dist/* followed by !dist/keep.txt.

You don't have to write it from scratch, either. GitHub's github/gitignore repo collects ready-made templates for dozens of languages and frameworks such as Node, Python, .NET and Java. The template you pick from the "Add .gitignore" list when you create a new repo on GitHub comes from this collection. Take a template as a starting point, then trim or extend it for your project.

As the rules pile up and a file is ignored, or not ignored, in a way you don't expect, don't guess; ask Git:

$ git check-ignore -v debug.log dist/bundle.js node_modules/left-pad/index.js
.gitignore:8:*.log	debug.log
.gitignore:5:dist/	dist/bundle.js
.gitignore:2:node_modules/	node_modules/left-pad/index.js
$ git check-ignore -v .env.example
.gitignore:13:!.env.example	.env.example

For each file, -v prints which file the rule is in, its line number and the pattern itself. The line for .env.example starts with !; in other words the file is not ignored, because the last matching rule brings it back in. In a big project with nested .gitignore files and a global file, this command saves hours.

An Already Tracked File: git rm --cached

Now for the opening question. .env was committed last week, and it is in .gitignore as well:

# No output for a tracked file
$ git check-ignore -v .env
$ git check-ignore -v --no-index .env
.gitignore:11:.env	.env
# After changing the key in .env
$ git status -s
 M .env
?? .DS_Store
?? .env.example
?? .gitignore

.env is listed with " M" even though it is in .gitignore. That is the answer to the opening question: .gitignore only applies to untracked files. Once a file is tracked, Git keeps tracking it, and the line in .gitignore changes nothing. That is why git check-ignore prints nothing for a tracked file; when we ask with --no-index, we see that the rule does match.

The fix is to stop tracking the file while leaving it on disk:

$ git rm --cached .env
rm '.env'
$ git status -s
D  .env
?? .DS_Store
?? .env.example
?? .gitignore
$ ls -1a
.
..
.DS_Store
.env
.env.example
.git
.gitignore
LICENSE.txt
README.md
debug.log
dist
node_modules
src

--cached means "remove it from the staging area only, don't touch the working tree". The D is in the left column: .env will leave the repo with the next commit. ls -1a shows that the file is still on disk. Had you forgotten --cached and run git rm .env, the file would have been deleted from disk too. After this commit, .env becomes untracked and .gitignore finally does its job.

One thing does not change, though: the key is still inside last week's commit. As we saw in the previous post, commits are snapshots, and that snapshot contains .env. If the repo was pushed, consider the key burned and generate a new one right away. There are tools that rewrite history, but in a shared repo the old copy may survive in every clone, so the real security measure is replacing the key.

Global Excludes: Noise That Belongs to You

.DS_Store is about your operating system, not about the project. Your teammate on Windows doesn't have such a file; they have the leftovers of their own editor instead. Rather than writing everyone's personal noise into every project's .gitignore, you give Git a global ignore file for your own machine:

$ git config set --global core.excludesFile '~/.gitignore_global'
$ git config get core.excludesFile
~/.gitignore_global
# After writing a .DS_Store line into ~/.gitignore_global
$ git status -s
D  .env
?? .env.example
?? .gitignore

We wrote ~ in quotes; Git expands it to the home folder itself. Even without the setting, Git looks at ~/.config/git/ignore by default, and putting the file there does the same job. There is also .git/info/exclude: an ignore file that applies only to that repo and only to you, and is never committed. It is ideal for your own experimental scripts.

Now we can commit. The removal of .env and the two new files go into the same commit:

$ git add .gitignore .env.example
$ git status -s
D  .env
A  .env.example
A  .gitignore
$ git commit -m "Ignore build output, logs and local settings"
[main 6d89b81] Ignore build output, logs and local settings
 3 files changed, 14 insertions(+), 1 deletion(-)
 delete mode 100644 .env
 create mode 100644 .env.example
 create mode 100644 .gitignore

git mv: Moving and Renaming

Let's rename the license file to the more common convention:

$ git mv LICENSE.txt LICENSE
$ git status -s
R  LICENSE.txt -> LICENSE
$ git commit -m "Rename LICENSE.txt to LICENSE"
[main 1634232] Rename LICENSE.txt to LICENSE
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename LICENSE.txt => LICENSE (100%)

R stands for rename. The 100% in the commit output says the content did not change at all. The interesting part is that Git does not store anything called a rename. The commit's tree has the new name and not the old one, and that's it. When Git sees the same or very similar content disappear under one name and show up under another, it presents that as a rename. So if you move the file in Finder or with mv and then run git add -A, you get the same R. git mv just turns two steps into one.

Line Endings with .gitattributes

In the installation post we said that core.autocrlf is a setting on each developer's own machine. If half the team is on Windows and someone's setting differs, commits start arriving in which every line of a file seems to have changed. .gitattributes puts this rule inside the repo:

* text=auto
*.sh text eol=lf
*.png binary

* text=auto means "for every file you detect as text, store its line endings as LF in the repo". *.sh text eol=lf forces shell scripts to come out with LF on every machine; a script with CRLF will not run on Linux. *.png binary tells Git not to treat these files as text and not to try to show a diff for them.

$ git add .gitattributes
$ git ls-files --eol
i/lf    w/lf    attr/text=auto        	.env.example
i/lf    w/lf    attr/text=auto        	.gitattributes
i/lf    w/lf    attr/text=auto        	.gitignore
i/lf    w/lf    attr/text=auto        	LICENSE
i/lf    w/lf    attr/text=auto        	README.md
i/lf    w/lf    attr/text=auto        	src/books.js
i/lf    w/lf    attr/text=auto        	src/cart.js
$ git commit -m "Normalize line endings"
[main e351c69] Normalize line endings
 1 file changed, 3 insertions(+)
 create mode 100644 .gitattributes

git ls-files --eol gives three pieces of information for each file: i/ is the line ending in the staging area, w/ in the working tree, and attr/ the rule that applies. Everything here is LF. If you add this file to an older repo that contains CRLF files, git add --renormalize . re-adds the existing files to the staging area according to the new rule; then you finish the switch with a single commit.

Common Pitfalls

The first pitfall is the opening question: adding a tracked file to .gitignore and thinking you are done. First git rm --cached, then commit. To untrack a whole folder, you need git rm -r --cached logs/.

The second is thinking the problem is solved once a secret is untracked. It is still in the earlier commits. A key that was pushed must always be rotated.

The third is trying to bring back a file inside an ignored folder with !. The exception won't work unless you write dir/* instead of dir/. When in doubt, check with git check-ignore -v.

The fourth is filling the project's .gitignore with personal files. .DS_Store, Thumbs.db or editor folders are about your machine; they belong in the global excludes file. Keep the project .gitignore for files the project itself produces.

A Small Experiment

★☆☆ Add a folder called coverage/ with a file in it to bookstore. Ignore it by adding a line to .gitignore, and confirm with git check-ignore -v which line catches it.

★★☆ While still ignoring the rest of the dist/ folder, get dist/README.md into the repo. First try adding !dist/README.md to the end of .gitignore. Use git status -s --ignored to see why your first attempt doesn't work, then fix the pattern.

★★★ Create a new config.local.json and commit it, then add it to .gitignore and change the file. See the opening situation for yourself with git status, untrack the file with git rm --cached and commit. Finally, check with git log --stat that the file is still in the old commit.

The repo's history is getting longer. In the next post we learn to move around in it: the --oneline, --graph, -p, --stat, --author, --since, --grep and -S options of git log, git show, git blame, which shows who wrote what line by line, and aliases for the commands you use often.