Historian: Because Please Stop Deleting My Bash History

For those on zsh I have something similar [1-2]. It hooks to zshaddhistory and stores the command, running time, CWD, hostname and exit status in a sqlite database, and provides a simple query command.<p>With a git merge driver [3] the history database can be kept in source control and shared between hosts.<p>Queries look a bit like<p><pre><code> $ histdb blah time ses dir cmd 09&#x2F;02 24 ~ ogr2ogr temp&#x2F;blah.shp &#x27;WFS:http:&#x2F;&#x2F;environment.data.gov.uk&#x2F;ds&#x2F;wfs?SERVICE=WFS&amp;INTERFACE=ENVIRONMENTWFS&amp;VERSION=1.0.0&amp;LC=3000000000000000000000000000000&#x27; 09&#x2F;02 24 ~&#x2F;temp cd blah.shp 15&#x2F;02 146 ~&#x2F;.emacs.d git commit -am &quot;blah&quot; 22&#x2F;02 175 ~ test=&quot;asdf&#x2F;blah.shp&quot; 09:48 743 ~&#x2F;.histdb hist blah </code></pre> The sess column here is a unique (per-host) session number which means it can recreate any transcript with a suitable query; if I ran histdb -s 24 it would produce the whole session containing the top two results above, including directory history.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;larkery&#x2F;zsh&#x2F;blob&#x2F;master&#x2F;sqlite-history.zsh" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;larkery&#x2F;zsh&#x2F;blob&#x2F;master&#x2F;sqlite-history.zs...</a> [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;larkery&#x2F;zsh&#x2F;blob&#x2F;master&#x2F;self-insert-override.zsh" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;larkery&#x2F;zsh&#x2F;blob&#x2F;master&#x2F;self-insert-overr...</a> [3] <a href="https:&#x2F;&#x2F;github.com&#x2F;larkery&#x2F;zsh&#x2F;blob&#x2F;master&#x2F;histdb-merge.zsh" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;larkery&#x2F;zsh&#x2F;blob&#x2F;master&#x2F;histdb-merge.zsh</a>

&gt; The last few versions of MacOS have introduced random fires which burn down your .bash_history, leaving this occasional sad, sad result:<p>$ wc -l .bash_history 0<p>I think there is some context missing here... Why is your .bash_history getting cropped? Was there a change in some default configuration of your Terminal app on update perhaps?<p>On a side-note I really think fish-shell really got history-recall and completion right, which is really the one thing that keeps me using it even if I still script in bash.

I&#x27;m a programmer and an historian. So the title of this entry gave me a wrong first impression: there are more people like me! I&#x27;m not alone! Imagine the disappointment... ;)

Something that has been serving me well over the years:<p>export PROMPT_COMMAND=&#x27;if [ &quot;$(id -u)&quot; -ne 0 ]; then echo &quot;$(date &quot;+%Y-%m-%d.%H:%M:%S&quot;) $(pwd) $(history 1)&quot; &gt;&gt; ~&#x2F;.logs&#x2F;bash-history-$(date &quot;+%Y-%m-%d&quot;).log; fi&#x27;<p>Creates a daily log file of every command entered. Searching for anything is a simple grep command.

What an interesting contrast in attitudes toward privacy:<p>He makes an extraordinary effort to maintain his bash history whereas I jump through hoops to delete my history (on logout using a .bash_logout file).<p>I also set my browser to maintain no history, disable saving of chats in everything, delete caches wherever possible, and generally wish that apps and programs had an option to save <i>zero</i> information on exit!<p>If I want &quot;history&quot; beyond a session, I&#x27;ll explicitly make a shell alias for a command that I want to remember, or make bookmark for a site I want to revisit, etc.

<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=10695305" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=10695305</a><p>^ In this comment I explained how I just have all my bash history logged directly into SQLite.

Shameless plug, my version of YABHA (yet another bash history app): <a href="https:&#x2F;&#x2F;github.com&#x2F;andmarios&#x2F;bashistdb" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;andmarios&#x2F;bashistdb</a><p>Written in go, uses sqlite and can work locally or remotely. In the latter case you can send history from many accounts and search globally. Transport encryption is based on NaCl. One binary serves all roles.<p>Doesn&#x27;t store other information than command line and time but this is what I need most of time.<p>The main pain point behind its creation was searching in history. There is a global flag (search in all accounts, -g), the wild card is SQL&#x27;s wild card (%). Less frequently used but very helpful, content search (-A, -B, -C, like grep). Also regexp is supported though very rarely is needed.

I&#x27;ve been curating my .bash_history for years; it&#x27;s a great way to recall all those contorted commands which come in handy every year or so:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;l0b0&#x2F;tilde&#x2F;blob&#x2F;master&#x2F;.bash_history" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;l0b0&#x2F;tilde&#x2F;blob&#x2F;master&#x2F;.bash_history</a><p>Git GUI plus a simple Makefile target to sort it is all the tooling necessary:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;l0b0&#x2F;tilde&#x2F;blob&#x2F;b16794f2ff209a6bc64813f0218168cca34e603f&#x2F;Makefile#L94" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;l0b0&#x2F;tilde&#x2F;blob&#x2F;b16794f2ff209a6bc64813f02...</a>

When I was first programming and into UNIX it was the norm amongst those I was learning from that you&#x27;d symlink your history file to &#x2F;dev&#x2F;null<p>This had a few advantages which might not be immediately apparent. The most obvious is that you remove the risk of leaking private or sensitive data.<p>The second is that searching your history is an anti-pattern. If there is a command you can&#x27;t remember, or if there is a group of commands you need to accomplish a task, you should be writing shell scripts<p>My ~&#x2F;bin folder has 104 files in it[0] and is in git, is available wherever I log in, whichever tty I have open and has a much better taxonomy than &#x27;search for part of command&#x27; based on doing tasks. The same can&#x27;t be said for history files.<p>[0] at its peak when I was a BSD and RHEL user it would have been more files but i&#x27;ve recently switched to Alpine.

If this article appeals to you, here&#x27;s a few little items that you may find handy (cf. bash(1)):<p><pre><code> $ shopt -s histappend $ export HISTTIMEFORMAT=&quot;%F %T: &quot; $ export HISTCONTROL=ignoreboth $ export HISTFILESIZE=-1 $ export HISTSIZE=-1 </code></pre> Also, maybe you should consider backing up ~&#x2F;.bash_history occasionally?

Odd that your history is being cleared out, it&#x27;s never happened to me.<p>I always keep my own history of every command I ever run. I&#x27;ve moved this from computer to computer. So I can look back years in the past and see what I&#x27;ve done.<p><pre><code> case &quot;$TERM&quot; in xterm*|rxvt*) PROMPT_COMMAND=&#x27;echo -ne &quot;\033]0;${USER}@${HOSTNAME}: ${PWD&#x2F;$HOME&#x2F;~}\007&quot;; echo `date +&quot;%b %e %Y %H:%M:%S&quot;` $? \&quot;`history 1|cut -c7-`\&quot; in `pwd` &gt;&gt; ~&#x2F;.audit&#x27; ;; *) ;; esac</code></pre>

It seems like Bash history was made for a time that is long gone, and has not been fixed to work in a modern environment. I have tons of shells open, and I cannot promise to close them carefully and sequentially.<p>Instead I found this to add to my .bashrc:<p><pre><code> pcmd=&#x27;if [ &quot;$(id -u)&quot; -ne 0 ]; then echo &quot;$(pwd) $(history 1)&quot; &gt;&gt; ~&#x2F;.logs&#x2F;bash-history-$(date &quot;+%Y-%m-%d&quot;).log; fi&#x27; PROMPT_COMMAND=&quot;$PROMPT_COMMAND;$pcmd&quot; </code></pre> It keeps all commands from all open shells in dated log files that I can grep through when Ctrl+R fails me.<p>Just be aware that if someone hacks into your PC they can see every command you have ever run, including any passwords accidentally pasted.

For anyone wondering, the correct way to do that cowsay command is `figlet moo |cowsay -n`

Last time my .zsh_history was randomly emptied on me I was so so stranded - I felt helpless.<p>I&#x27;ve now doubled-down on it and actually store lots of todo notes or receipt numbers or random snippets like that in my shell history as comments. It&#x27;s just always there :-)<p>And fortunately btrfs snapshots (and one time, a backup from a laptop migration...) have saved me form losing it for a long time now :-)<p>Shell history is awesome, I&#x27;m always really intrigued when I see someone like a sysadmin who doesn&#x27;t use shell history very well (read, at all...)

Alternatively, you can use ZSH and not lose history, even better, have history work simultaneously in multiple terminal windows simultaneously without closing the window to save&#x2F;read the history of other windows. <a href="http:&#x2F;&#x2F;ohmyz.sh&#x2F;" rel="nofollow">http:&#x2F;&#x2F;ohmyz.sh&#x2F;</a>

Alternatives to this implementation:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;barabo&#x2F;advanced-shell-history" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;barabo&#x2F;advanced-shell-history</a><p><a href="https:&#x2F;&#x2F;github.com&#x2F;tkf&#x2F;rash" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;tkf&#x2F;rash</a><p>both are fairly featureful and capture additional metadata about the commands run<p>it is also possible to use similar tools to log your bash history to mysql, or to syslog, which may be preferable for auditing purposes.

I&#x27;ve gradually come up with a way too elaborate system for maintaining and sharing bash histories locally. I run tmux with several long-running bash prompts and not only I want to append each shell&#x27;s history into a log incrementally but also to share history between the running shells.<p>I have a function as my prompt command that does:<p>- dump the current in-memory history into a separate logfile every few minutes timestamped (using HISTTIMEFORMAT)<p>- clear the local in-memory history and reload in-memory history from unique history lines from all stored log files, using the most recent timestamp on each<p>Once there are more than N log files, one of the shells does a similar uniquefying compation step for N log files and writes the output into one new log file. It uses &#x27;mv&#x27; to do the compation atomically by moving the affected log files away into a temporary directory for compaction.<p>If other shells were to update their logs during the compaction those additions would just be saved as normal and compacted in the next cycle. There are always duplicate history lines in the logs to keep things fast so there&#x27;s no chance for a race condition: no command line in the history can get accidentally deleted and any duplicates are just removed during reading.<p>As a final step I atomically copy the latest compacted history into .bash_history as a backup. The real data lives in my logs anyway but a single .bash_history is easier to backup: it might be a few lines short but it&#x27;s always mostly up-to-date.<p>Looking back at this, I reckon the current Bash history implementation surely can&#x27;t be ideal.

Wow, cool! How does this handle the case of multiple shells writing to the history at the same time? I have yet to find a safe way to preserve all that.

I really like how the search feature shows the frequency of each entry. I&#x27;ve thought before it would be cool to see the most used commands of some of the more experienced folks I work with, though ideally it would be slightly more generic than exact commands - i.e. Different arguments would count for the same command.

Or, you know, put out the &quot;random fires that burn down your history&quot;

I like this. It looks like a much better fleshed out version of a one-off python script I use to search bash history - it even coincidentally has the same name so it&#x27;ll fit right into the muscle memory :D<p><a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;43019250&#x2F;made-quick-script-to-search-bash-history-how-can-i-make-it-more-robust&#x2F;" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;43019250&#x2F;made-quick-scri...</a>

HISTFILE=~&#x2F;bash_history&#x2F;$(date +%Y-%m) with h() { grep -a $@ ~&#x2F;bash_history&#x2F;* }<p>Work for me... I don&#x27;t lose bash history.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;kaihendry&#x2F;dotfiles&#x2F;blob&#x2F;master&#x2F;.bashrc#L34-L47" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;kaihendry&#x2F;dotfiles&#x2F;blob&#x2F;master&#x2F;.bashrc#L3...</a>

I love this idea. It&#x27;s not important for production &#x2F; env that are managed through tooling, but the &quot;randomish&quot; nature of losing history on dev&#x2F;stage is one of those, admittedly minor, annoyances that this would solve.<p>Great idea, thanks for sharing.

I&#x27;ve been using a variant of this: <a href="https:&#x2F;&#x2F;debian-administration.org&#x2F;article&#x2F;543&#x2F;Bash_eternal_history" rel="nofollow">https:&#x2F;&#x2F;debian-administration.org&#x2F;article&#x2F;543&#x2F;Bash_eternal_h...</a> which adds all commands to a file ~&#x2F;.bash_eternal_history<p>Plus a script named &quot;ehist&quot; in my path which contains &quot;cat ~&#x2F;.bash_eternal_history&quot; -- this enables searches with &quot;ehist | grep git&quot;<p>It also handles the case of multiple shells writing to a history file.

Some sites require the periodic deletion of .bash_history and .history files as a security policy, because they&#x27;re quite likely to contain passwords and other secrets in them.

I wish there were a tool to share .bash_history across an organization. This would make onboarding so much easier. It would also help to disseminate shell tricks.

I think a lot of the problems linked to bash history could be solved elegantly by the use of a daemon to exchange information between concurrently opened shells.

On a sidenote, how do I tell bash to immediately commit to history? I often close iterm tabs with cmd-w which seems to make my bash forget to write history.

How can we store history of many users [with screen sessions] on another server - as a simple history keeping and monitoring thing? I looked into auditd but did not like the logging format (maybe there is just a nice web front-end missing that interprets the details) - a simple history would be enough.<p>Next todo was to look if we could send the history to some syslog server - do you have any experiences with a similar solution?

I use my shell history to pickup past command invocations, but at some point, you have to clean things up and create a script. The script could be for repetitive tasks or help tame complicated command line syntax. The nmap, curl and groff utilities are a few of my favorite examples of commands that have too many options.

my bash history gets deleted whenever I inadvertently type my password and hit enter at the wrong prompt. Type sudo 30 time a day, its gonna happen. Yes I know my history has permissions protection, but still. It is plain text. I don&#x27;t want to have to look for all the places it get copied too with these solutions.

As a pen tester who frequently finds passwords and other sensitive data in shell history files[1], I fully support this effort to make my job even easier :).<p>[1] Either inline in mysql-type commands, or when users accidentally paste&#x2F;type their password in when the system isn&#x27;t expecting it.

If you touch ZSH (oh my zsh, if that makes a difference) once you will never ever want to use bash again.

I use git&#x2F;gitlab in a private repo to keep track of my zsh profile and history. Stupid like a fox!

What are the random fires? please provide an example, i like to control my HISTFILE<p>edit: FIRE == OSX specific

&quot;<i>In its nascent form, a script backs up your .bash_history into a SQLite database that sits in your home directory as well.</i>&quot;

I only save about 40 lines of shell history. If I find myself repeating something I make a script or alias.<p>Root account never saves history.

I think this is nice. I use tmux every day and this will help me to keep my bash history in my multiple tabs.

ctrl-r is great, but i&#x27;d like to also delete unwanted results and favourite some and label them.

Are there any bash history systems which also store the directory the command was issued in?

didn&#x27;t even know that&#x27;s the case, is zsh affected by this mac os bug?

My hand made script doing the same thing is also called &quot;hist&quot;.

TIL about figlet and cowsay.<p>Thanks HN!

fish