1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
|
QuickStart:
On an Ubuntu 14.04+/Debian Wheezy+ machine or VM, run:
$ make
This will check all of your prerequisites and tell you what you need to
do to get started to build the stable TBB.
If everything checks out OK, it will begin downloading dependencies, and then
start the build process to produce localized Linux bundles, followed by
Windows bundles, followed by Mac bundles.
To check your build results against the official builders, run:
$ make match
By default, the Makefile wraps input downloads in 'torsocks'. This is done
to allow unofficial secret verifiers to remain secret by default. To download
inputs without torsocks, run:
$ make TORSOCKS=
By default, 'make' will also clear out any previous partial builds. See the
"Partial Rebuilds" section for information on performing partial rebuilds
without erasing previously built components.
If you would also like to download fresh copies of your inputs before a
build, run 'make distclean' (note that timestamps and versions of downloaded
source input files are checked every time you run 'make' by default so
distclean should not normally be needed).
To set the number of make processes and virtual CPUs to use inside the VMs
to four (the default is two), use:
$ export NUM_PROCS=4
To set the amount of RAM to use inside the VMs to 8000 MiB (the default is
4000 MiB), use:
$ export VM_MEMORY=8000
Detailed Explanation of Scripts:
This directory is a wrapper around our modified version of Gitian, and has
several helper scripts to make things easier.
0. Makefile: The main Makefile. It has the following commands:
- all: The default. It calls the 'clean', 'prep' and 'build' rules
- prep: Check OS prerequisites and download source dependency inputs
- build: Build localized bundles for Linux, Windows, and Mac
- clean: Remove prior partial build stages (see 'Partial Rebuilds' below
for the usage of clean-* commands)
- vmclean: Remove VM base images
- distclean: Remove source dependency inputs, and run 'clean' and 'vmclean'
- sign: Sign your build output and uploads it to people.torproject.org
- match: Check your build output against public signed hashes
- hash: Create the SHA256 sums
- incrementals: Create the incremental update (.mar) files
- update_responses: Create .xml and .htaccess files for the updater
- signmars: Create the signatures for the update (.mar) files
To build beta/alpha/nightly bundles, alternate targets are provided:
- nightly: The equivalent to the 'all' rule for nightly packages
- alpha: The equivalent to the 'all' rule for alpha packages
- prep-nightly: The equivalent to the 'prep' rule for nightly packages
- prep-alpha: The equivalent to the 'prep' rule for alpha packages
- build-nightly: The equivalent to the 'build' rule for nightly packages
- build-alpha: The equivalent to the 'build' rule for alpha packages
- sign-nightly: The equivalent to the 'sign' rule for nightly packages
- sign-alpha: The equivalent to the 'sign' rule for alpha packages
- match-nightly: The equivalent to the 'match' rule for nightly packages
- match-alpha: The equivalent to the 'match' rule for alpha packages
- hash-nightly: The equivalent to the 'hash' rule for nightly packages
- hash-alpha: The equivalent to the 'hash' rule for alpha packages
- incrementals-alpha: The equivalent to the 'incrementals' rule for
alpha packages
- update_responses-alpha: The equivalent to the 'update_responses' rule
for alpha packages
- signmars-nightly: The equivalent to the 'signmars' rule for nightly packages
- signmars-alpha: The equivalent to the 'signmars' rule for alpha packages
1. check-prerequisites.sh: This script checks if your system is capable of
running Gitian, and if it is not, it tells you what you need to do.
It is called by 'make prep'.
2. fetch-inputs.sh: This script fetches all of our source dependencies from
the Internet and downloads them into ../../gitian-builder/inputs.
After you run this script, you should be able to complete the rest of your
build offline.
3. versions: This scriptlet contains version definitions for the source
dependencies, the git tags to use, and the Tor Browser release version.
4. verify-tags.sh: This script verifies the signatures on git tags from
the versions file. It is only run if VERIFY_TAGS is set in the versions
file.
5. descriptors: In the descriptors directory, you will find the Gitian
descriptors for Linux, Windows, and Mac. There are five descriptors for
each platform: One to build the utilities needed by other descriptors,
one to build Tor, one to build Tor Browser, one to build the Pluggable
Transports and one to bundle everything together. Each descriptor is run
from a fresh VM.
6. mkbundle-linux.sh: This script is a wrapper around Gitian's gbuild to call
the appropriate descriptors to build 32 and 64 bit Linux bundle. It also will
create build VM images for you if you haven't done that before.
7. mkbundle-windows.sh: This script is a wrapper around Gitian's gbuild to
call the appropriate descriptors to build a Windows bundle. It also will
create build VM images for you if you haven't done that before.
8. mkbundle-mac.sh: This script is a wrapper around Gitian's gbuild to
call the appropriate descriptors to build a 32bit Mac bundle. It also
will create build VM images for you if you haven't done that before.
9. record-inputs.sh: This script records all of the hashes and bundle
inputs for inclusion in the 'Docs/sources' subdirectory of the bundles
themselves.
10. hash-bundles.sh: This script generates a 'sha256sums-unsigned-build.txt'
file in sorted, reproducible order.
11. check-match.sh: This script checks your 'sha256sums-unsigned-build.txt'
file against any signed, published builds.
12. upload-signature.sh: This script signs and uploads your
'sha256sums-unsigned-build.txt' file (for use if you are an official
builder).
13. signmars.sh: This script generates the signatures on the update (.mar)
files. It expects an nssdb directory, containing the key, in the same
directory (i.e. tor-browser-bundle/gitian where it is located, too) and
a certificate with "marsigner" as nickname. Both the default nssdb
directory and the default certificate nickname can get overwritten by
exporting 'NSS_DB_DIR' and 'NSS_CERTNAME' pointing to the directory and
the nickname to be used instead.
Partial Rebuilds:
The mkbundle shell scripts each have five Gitian stages:
1. Build support utilities (gitian-utils.yml)
2. Tor (gitian-tor.yml)
3. Tor Browser (gitian-firefox.yml)
4. Pluggable Transports (gitian-pluggable-transports.yml)
5. Addon and bundle package assembly (gitian-bundle.yml)
Each stage leaves build artifacts in ../../gitian-builder/inputs/ for use
in later stages. If the artifact files for a given stage are present, that
stage is skipped by a 'make resume'.
No other dependency checking on build stages is performed.
If you wish to rebuild only a given stage, there are several 'make clean-'
targets to remove the artifacts for that stage. Running a subsequent
'make resume' after that will rebuild only those targets that have been
cleaned, and then re-package everything again as well.
Note that if any of the source inputs, or the Ubuntu buildchain tools, or
the build tools' dependencies upgrade, your Tor and Tor Browser inputs will
no longer match those produced by someone else's build from a fresh build
that downloaded those newer input, dependency, and toolchain packages.
This means for official builds, it is a good idea to always run a full
'make clean' first, or just run 'make'.
Known Issues and Quirks:
1. Sometimes, fresh VM installs can fail. If your build hangs while trying
to connect to the target or during the SSH banner, it might be worth
running 'make vmclean && make resume' to clean your VM images, but still
resume the build process as the component you left off at. Remember,
'make' by itself runs a clean rule that will wipe your previous builds by
default.
If you have already completed a Linux build, you may run into situations
where 'make vmclean' causes the rebuild of two VMs in a row.. This might
trigger weird bugs in python-vm-builder.. To rebuild only one set of VMs,
use either 'rm ../../gitian-builder/*precise*' (to remove the Windows/Mac
VMs) or 'rm ../../gitian-builder/*wheezy*' (to remove the Linux VMs).
You probably want to make sure you have no stray qemu processes before
rebuilding the VMs or starting a new build, too. 'killall qemu-kvm' is
your friend.
Once you get a working set of base VMs (in ../../gitian-builder/base-*),
you should probably copy them somewhere safe if you want to avoid VM
creation hangs in the future (or help us write a wrapper script that
tests VMs and re-runs the VM creation step if they don't boot).
2. If you use git branches for any repos instead of tags (for example, for
a development or nightly build), fresh commits will need to be
merged manually (or better git-fu is needed in ./fetch-inputs.sh, as
it is currently meant to deal with tags only).
3. Running more than one instance of Gitian at a time is not supported.
4. Related: If you perform a fresh Gitian checkout for purposes of
verification, be sure to kill any stray qemu VM processes before starting
the new build (because the new Gitian checkout will not have the PID or
SSH keys of the previous instances' VM, and VM startup will either hang
or prompt you for an SSH password).
Diagnosing Problems:
During a running build, you can tail logs in real time in
../../gitian-builder/var/*.log
Upon failure, logs of any failed component builds are relocated to
../../gitian-builder/*fail*.log. This is to prevent subsequent builds
from destroying failure information.
|
