Webapp.io has extended & improved Docker’s caching model for use in CI.
Consider the following Layerfile:
FROM vm/ubuntu:18.04 COPY . . RUN sleep 20 && cat file1 RUN sleep 20 && cat file2
In this case, we’ll make snapshots after each line and map which files were read back to the snapshots. This means:
Here are the major differences between Layerfiles and Dockerfiles for use in CI:
COPYin webapp.io does not invalidate the cache when it runs, instead the files copied are monitored for read/write starting at that point. This means that
COPY . .is much more common in Layerfiles than Dockerfiles
COPY /file1 .or
COPY ../.. .) and inherit from other Layerfiles
In most CI providers and in Docker, you need to micromanage cache keys. The following Dockerfile and Layerfile are equivalent because we watch which files are read by each step:
FROM ubuntu:18.04 COPY package.json package-lock.json ./ RUN npm install COPY . . RUN npm run build
FROM ubuntu:18.04 COPY . . RUN npm install RUN npm run build
Instead of micromanaging COPY, you can simply copy the entire repository and we’ll load the bottommost layer from the cache which agrees with a commit’s changes.
Sometimes there are steps which will run repeatedly because their constituent files change often, usually source files. Consider this Layerfile:
FROM vm/ubuntu:18.04 RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - && \\ echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list && \\ curl -fSsL https://deb.nodesource.com/setup_12.x | bash && \\ apt-get install nodejs yarn MEMORY 2G ENV NODE_OPTIONS=--max-old-space-size=8192 COPY package.json ./ CACHE /usr/local/share/.cache/yarn RUN npm ci
In this case, unless you change
package.json, the default webapp.io cache will
skip the entire pipeline after every push.
CACHE directive only acts to speed up the
npm ci step in this case.
CACHE will “leak” state across runs, so it might allow one run to
break all following ones until someone force-retries without caches. To avoid
this problem, only cache stateless directories (which usually contain “cache” in
Some other examples: