PER MØLDRUP-DALUM

Process < Input > Output # About

Tracking Time

22 March 2024

These days, I'm trying to implement the final system for tracking how much time I spend on different types of tasks and projects to improve my ability to estimate the time needed for these tasks and better plan my time, as time is a very scarce resource. Also, in my job, I'm required to report my time for a few international projects funded by the EU.

I've always been a fan of automation and love spending way too much time implementing some small script that automates a task — and then forgetting that I wrote it ;-)

xkcd: Is It Worth the Time?
xkcd: Is It Worth the Time?.

Now, I've been exposed to many different solutions for time tracking, from ancient 90's Windows applications, over Harvest and Toggl to Excel sheets. I've also explored the space of time-tracking applications on macOS, both GUI and CLI, and I’ve built too many myself. The one solution that I, over the years, have used most consistently is orgmode in Emacs (my preferred Emacs flavor is Mitsuharu Yamamoto's Mac port), but why I don’t choose that solution going forward is a story in itself which is waiting to be told...

Over the years, I've realized that the most important feature of time-tracking applications is the ease of adjustment. During any working day, things happen. Phone calls, people dropping by, an idea pops into your mind, a task drags out or is finished early, you forget what you're working on, bio-breaks, focus breaks, etc. So, no matter what or how well you plan, Helmuth von Moltke was right when he wrote "Kein Operationsplan reicht mit einiger Sicherheit über das erste Zusammentreffen mit der feindlichen Hauptmacht hinaus." (Moltke, 1892). In the business of software development, this is also related to The Mythical Man-Month (Brooks, 1995) and what could be seen as a kind of response to that book, The Agile Manifesto. You need to have a system that responds well to change and complexity.

Therefore, the time-tracking records must be easily adjustable using as few computer touches (keyboard, mouse, touchpad, screen) as possible. So no matter how proficient you are at vim-golf or mouse gestures, every touch counts.

Another very important part of such a system is to leverage the computer. Don’t spend time doing something that a computer would be better at. For example, I’ve never come across any requirement on having time records precise to the second. Still, you want every second to count, but to adjust your records to that granularity is very tiresome to you, but not to a computer. So, if you just make an effort to be precise within, say, five minutes, the computer can handle the rest by expanding or shrinking your records to fit each other, giving you 100% coverage during the work day within a precision of a few minutes.

So, a time-tracking system has three components: input and adjustment, storage, and reporting.

Input and adjustment

I’ve used my calendar for many things, and back in the day when calendars were small paper books, there were no limits to what they could be used for, even without going the full Filofax route. I’ve literally got dozens of paper calendars stored in boxes that reflect my life via personal diary entries, general life observations, shopping lists, project management, appointments (do'h!), TV shows, movies, books, etc.

When digital calendars started taking over, I quickly jumped on board and bought a Palm III device, but ever since, I’ve been struggling in this digital domain with how to keep and recall what I used to use my paper calendars for.

So, I'm used to having a lot in my calendar, which, as I’m on macOS, is the Apple Calendar, and my calendar is always open and up to date.

I order to handle time tracking in general and across multiple applications, I need a system for connecting records across the platform. Therefore, I have a naming syntax for all tasks, projects, and categories to make entering time spent on a task or actual task categories easy. This syntax evolves, but as I keep my system computable-by-design (LINK), changes in syntax can easily be reflected in the calendar. The most recent syntax is inspired by the Johnny.Decimal system, though I’ve changed it quite a bit. I have all work projects and areas of responsibility listed on one level. This list is divided into categories by a naming convention that looks like the following. 

00-work
01-FORSK-<research project name>
01-FORSK-<research project name>
...
02-INFRA-<infrastructure project>
02-INFRA-<infrastructure project>
...
03-SNARFU-<work groups, steering groups, committees, councils, boards, etc>
03-SNARFU-<work groups, steering groups, committees, councils, boards, etc>
...
04-OUTREACH
05-workshops and conferences
06-VEJL-<guidance and consultancy>
06-VEJL-<guidance and consultancy>
...
07-PROJ-<general project>
07-PROJ-<general project>
...
09-REFS-<reference and library>
09-REFS-<reference and library>
...
11-TRAVEL-
13-APPLICATION- 
21-INTERN-
22-META-GTD-
22-META-SYS-DMS-
23-RD-
24-LEARN-

For most of the above things, I only need to record the number and description, so entering a record into the calendar is as simple as pressing ⌘N, writing e.g., 01-FO, and the completion engine of Apple Calendar suggests “01-FORSK”, and if the information is available, all the research projects I have previously registered time on would also be available for completion. If I spell out the complete name, I can also use the calendar’s understanding of natural language to input estimated time usage, e.g., “01-FORSK from 8:15 to 11:30,” and I’m done registering the task.

Then, when things change, I just drag the end points of the event in the calendar to match what happened. There is no fiddling with time widgets.

If I then need to change 01-FORSK into 001-FORSK, I can use an Applescript to that end. By the way, this is my computational-by-design philosophy (which is another story to be written). 

use AppleScript version "2.4"
use scripting additions

tell application "Calendar"
    tell calendar "Tidsregistrering" -- the calendar used for time tracking
        set allEvents to every event whose start date is greater than or equal to date "1/1/2024" and summary is "01-FORSK"
        repeat with i from 1 to length of allEvents
            set thisEvent to item i of allEvents
            tell thisEvent
                set summary of thisEvent to "001-FORSK"
            end tell
        end repeat
    end tell
end tell

Storage

This I get for free, as it's handled by Apple Calendar and synchronized to all my devices. Also, whenever a report is created, a backup of the data is stored in a safe location — just to be safe and not sorry.

Reporting

This is split into two distinct parts: data extraction and the actual reporting.

Extract data

To extract data from Apple Calendar, the first choice would be to write some Applescript code like below 

use AppleScript version "2.4" -- Yosemite (10.10) or later
use scripting additions

set outText to "Summary" & "    " & "start" & " " & "end" & "
"

set dateString to "01-01-" & ((current date)'s year as string)
set startDate to date dateString

tell application "Calendar"
    tell calendar "Tidsregistrering"
        set allEvents to every event whose start date is greater than or equal to startDate
        repeat with i from 1 to length of allEvents
            set thisEvent to item i of allEvents
            tell thisEvent
                set sd to (start date as «class isot» as string)
                set ed to (end date as «class isot» as string)
                set outText to outText & summary & "    " & sd & "  " & ed & "
"
            end tell
        end repeat
    end tell
end tell

--set utf8Data to (outText as «class utf8»)

--set filePath to (path to home folder as text) & "myfile.txt"

set y to do shell script "date +%Y"
set m to do shell script "date +%m"
set d to do shell script "date +%d"

--set fileName to ("tidsregistreringer-2024-" & day of (current date) as string) & ".tsv"
set fileName to ("tidsregistreringer-" & y & "-" & m & "-" & d) & ".tsv"


set theFile to (((path to home folder) as string) & "Documents:" & fileName)
writeTextToFile(outText, theFile, true)
return theFile

(*
-- Save the UTF-8 data to a file
set fileRef to open for access filePath with write permission
set eof of fileRef to 0
write utf8Data to fileRef starting at 0
close access fileRef
*)
on writeTextToFile(theText, theFile, overwriteExistingContent)
    try
        
        -- Convert the file to a string
        set theFile to theFile as string
        
        -- Open the file for writing
        set theOpenedFile to open for access file theFile with write permission
        
        -- Clear the file if content should be overwritten
        if overwriteExistingContent is true then set eof of theOpenedFile to 0
        
        -- Write the new content to the file
        write theText to theOpenedFile starting at eof as «class utf8»
        
        -- Close the file
        close access theOpenedFile
        
        -- Return a boolean indicating that writing was successful
        return true
        
        -- Handle a write error
    on error
        
        -- Close the file
        try
            close access file theFile
        end try
        
        -- Return a boolean indicating that writing failed
        return false
    end try
end writeTextToFile

The problem with Applescript is that it is very slow. For example, the above code runs for minutes on my MacBook Air M2! Luckily, Ali Rantakari has created icalBuddy, which is very fast at querying the Calendar database. The code is released under the MIT License and available on GitHub at github.com/ali-rantakari/icalBuddy.

Now, icalBuddy hasn’t been updated since 2013, which makes me a bit worried, but apparently, Apple has not changed the underlying database structure for Calendar, which in itself is rather interesting. So, anyway, I’m using icalBuddy until it breaks. When it breaks, three options remain: Hope someone implements new code using the source at Gihhub, implement it myself, or go back to the slow-as-molasses Applescript code. The good thing is that this is not a threat to my overall system for time tracking.

So, as I was implementing this, I ran into another issue that made me learn something new about macOS, which was fun.

The Calendar on macOS is protected by system-wide privacy and security settings, and command-line utilities such as icalBuddy can not be approved for such access on their own. So what happens is that the application that handles the call to icalBuddy must be approved and added to the list of applications that have access to the calendar data, but this list cannot be controlled in any way by the user. The only way to get an application added is for it to ask macOS for permission, which then presents the user with a dialog asking for permissions.

Of course, I could have a two-step process for extraction and reporting, but I’m too lazy. Everything should be done in one step.

Then again, I could wrap the solution in a shell script that first invokes icalBuddy for export into a temporary file and then invokes the reporting application. Still, I want to do my work from within the reporting application. I know, my choice.

These days, I do most, if not all, of my data crunching in Mathematica, but the next steps apply in the same way for RStudio, Jupyter, VSCode, etc.

So, I kept getting error: No calendars. when trying to run icalBuddy from Mathematica: 

RunProcess[{"/opt/homebrew/bin/icalBuddy", "calendars"}]

(*
<|"ExitCode" -> 1, "StandardOutput" -> "", 
 "StandardError" -> "error: No calendars.
"|>
*)

I could also provoke the same error on the command line by removing the calendar permission for Terminal.

 ~ >: icalBuddy eventsToday
error: No calendars.

Then I came across several posts, e.g., Upcoming-iCal-Events/issues/9 where users of Übersicht were having the same kind of problems. Back when icalBuddy was created, the Reminders and Calendar applications from Apple were much more integrated than now, wherefore it makes sense for this dual permission issue.

The next step was to get Mathematica on the list of applications approved for access to the Reminders data. As said above, the macOS GUI offers no interface for adding (or removing) applications to this list.

Then I got the idea of just using some Applescript from within Mathematica that tries to access the Reminders store: 


s = "tell application \"Reminders\"
set mylist to list \"Your List Name\"
tell mylist
make new reminder at end with properties {name:\"hello\", due \
date:date \"7/10/2014 3:00 PM\"}
end tell
end tell";

RunProcess[{"osascript", "-e", s}]
(*
Out[•]= <|"ExitCode" -> 1, "StandardOutput" -> "", 
 "StandardError" -> 
  "43:64: execution error: Reminders modtog en fejl: Kan ikke hente \
list \"Your List Name\". (-1728)
"|>
*)

Execution of the RunProcess[…] asked permission to access the Reminders data store. Yes! Please! ...and then Mathematica was added to the Reminders list and everything works 🥳

This concludes this part. I now have a system for recording time usage, adjusting the records, and extracting those records for reporting.

Creating reports of time usage

W.I.P.

  1. Write code for extracting data
  2. Write code for backup of data
  3. Write code for adjusting records to fill the working hours
  4. Write code for validating the records
  5. Design reports
  6. Implement reports
  7. Automate
  8. Make into a Wolfram Package and publish

Bibliography

Moltke, Helmuth. Moltkes militärische Werke. Berlin: E.S. Mittler, 1892. Direct link to Internet Archive

Brooks, Frederick P. The Mythical Man-Month: Essays on Software Engineering. Anniversary ed. Reading, Mass: Addison-Wesley Pub. Co, 1995.

“Manifesto for Agile Software Development.” Accessed March 24, 2024. agilemanifesto.org.