aboutsummaryrefslogtreecommitdiffstats
path: root/www/projects.html
blob: e375eb9cc3d2c34e5f4fc6e4cba04c8a7d680ebe (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<link href="style.css" rel="stylesheet" type="text/css" />
<title>LLDB Projects</title>
</head>

<body>
    <div class="www_title">
      Project ideas for the <strong>LLDB</strong> Debugger
    </div>
    
<div id="container">
	<div id="content">
         <!--#include virtual="sidebar.incl"-->
		<div id="middle">
			<div class="post">
				<h1 class ="postheader">Projects</h1>
				<div class="postcontent">

					  <p>The following is a mostly unordered set of the ideas for improvements
                                            to the LLDB debugger.  Some are fairly deep, some would require less 
                                            effort.
					     </p>

					  <ol>
                                            <li>
                                              Speed up type realization in lldb.  
                                              <p>
                                                The type of problem I'm addressing here is the situation where you are 
                                                debugging a large program (lldb built with debug clang/swift will do) 
                                                and you go to print a simple expression, and lldb goes away for 30 seconds.  
                                                When you sample it, it is always busily churning through all the CU's in the
                                                world looking for something.  The problem isn't that looking for something in 
                                                particular is slow, but rather that we somehow turned an bounded search 
                                                (maybe a subtype of "std::string" into an unbounded search (all things with the 
                                                name of that subtype.)  Or didn't stop when we got a reasonable answer 
                                                proximate to the context of the search, but let the search leak out globally. 
                                                And quite likely there are other issues that I haven't guessed yet. 
                                                But if you end up churning though 3 or 4 Gig of debug info, that's going to be slow 
                                                no matter how well written your debug reader is...
                                              </p>
                                              <p>
                                                My guess is the work will be more in the general symbol lookup than in the DWARF 
                                                parser in particular, but it may be a combination of both.
                                              </p>
                                              <p>
                                                As a user debugging a largish program, this is the most obvious lameness of lldb.
                                              </p>
                                            </li>

                                            <li>
                                              Symbol name completion in the expression parser.
                                              <p>
                                              This is the other obvious lameness of lldb.  You can do:
                                              </p>
<code><pre><tt>
(lldb) frame var foo.b<TAB>
</tt></pre></code>
                                              <p>
                                                and we will tell you it is "foo.bar".  But you can't do that in the expression parser. 
                                                This will require collaboration with the clang/swift folks to get the right extension 
                                                points in the compiler. And whatever they are, lldb will need use them to tell the 
                                                compiler about what names are available.  It will be important to avoid the pitfalls 
                                                of #1 where we wander into the entire DWARF world.
                                              </p>

                                            </li>
                                            <li>
                                              Make a high speed asynchronous communication channel to replace the gdb-remote protocol.
                                              <p>
                                                All lldb debugging nowadays is done by talking to a debug agent.  We used the 
                                                gdb-remote protocol because that is universal, and good enough, and you have 
                                                to support it anyway since so many little devices & JTAG's and VM's etc 
                                                support it.  But it is really old, not terribly high performance, and 
                                                can't really handle sending or receiving messages while the process is 
                                                supposedly running.  It should have compression built in, remove the hand-built 
                                                checksums and rely on the robust communication protocols we always have nowadays,
                                                allow for out-of-order requests/replies, allow for reconnecting to a temporarily 
                                                disconnected debug session, regularize all of the packet formatting into JSON or 
                                                BSON or whatever while including a way to do large binary transfers.  It must be 
                                                possible to come up with something faster, and better tunable for the many 
                                                communications pathways we end up supporting.
                                              </p>
                                            </li>

                                            <li>
                                              Fix local variable lookup in the lldb expression parser.

                                              <p>
                                                The injection of local variables into the clang expression parser is 
                                                currently done incorrectly - it happens too late in the lookup.  This results
                                                in namespace variables & functions, same named types and ivars shadowing 
                                                locals when it should be the other way around.  An attempt was made to fix
                                                this by manually inserting all the visible local variables into wrapper function
                                                in the expression text.  This mostly gets the job done but that method 
                                                means you have to realize all the types
                                                and locations of all local variables for even the simplest of expressions, and
                                                when run on large programs (e.g. lldb) it would cause unacceptable delays.  And
                                                it was very fragile since an error in realizing any of the locals would cause
                                                all expressions run in that context to fail.  We need to fix this by adjusting
                                                the points where name lookup calls out to lldb in clang.
                                              </p>
                                            </li>
                                            <li>
                                              Fix the event handling/process control machinery to support calling SB & Commands 
                                              everywhere, and to support non-stop debugging
                                              <p>
                                                There is a fairly ad-hoc system to handle when it is safe to run SB API's and 
                                                command line commands.  This is actually a bit of a tricky problem, since we 
                                                allow access to the command line and SB API from some funky places in lldb. 
                                                The Operating System plugins are the most obvious instance, since they get 
                                                run right after lldb is told by debugserver that the process has stopped, but 
                                                before it has finished collating the information from the stop for presentation 
                                                to the higher levels.  But breakpoint callbacks have some of the same problems, 
                                                and other things like the scripted stepping operations and any fancier extension 
                                                points we want to add to the debugger are going to be hard to implement robustly 
                                                till we work on a finer-grained and more explicit control over who gets to control 
                                                the process state.
                                              <p>
                                                We also won't have any chance of supporting non-stop debugging - which is a useful
                                                mode for programs that have a lot of high-priority or real-time worker threads - 
                                                until we get this sorted out.
                                              </p>
                                            </li>

                                            <li>
                                              Finish the language abstraction and remove all the unnecessary C++/clang-based API's
                                              <p>
                                                An important part of making lldb a more useful "debugger toolkit" as opposed to a 
                                                C/C++/ObjC/Swift debugger is to have a clean abstraction for language support. 
                                                We did most, but not all, of the physical separation.  We need to finish that. 
                                                And then by force of necessity the API's really look like the interface to a C++ 
                                                type system with a few swift bits added on.  How you would go about adding a new 
                                                language is unclear and much more trouble than it is worth at present.  But if 
                                                we made this nice, we could add a lot of value to other language projects.
                                              </p>
                                            </li>

                                            <li>
                                              Add some syntax to generate data formatters from type definitions
                                              <p>
                                                Uses of the data formatters fall into two types.  There are data formatters for 
                                                types where the structure elements pretty much tell you how to present the data, 
                                                you just need a little expression language to express how to turn them into what 
                                                the user expects to see.  Then there are the ones (like pretty much all our 
                                                Foundation/AppKit/UIKit formatters) that use deep magic to figure out how the type 
                                                is actually laid out.  The latter are pretty much always going to have to be done by hand.
                                              </p>
                                              <p>
                                                But for the ones where the information is expressed in the fields, it would be great 
                                                to have a way to express the instructions to produce summaries and children in some 
                                                form you could embed next to the types and have the compiler produce a byte code 
                                                form of the instructions and then make that available to lldb along with the library.
                                                This isn't as simple as having clang run over the headers and produce something from the 
                                                types directly.  After all, clang has no way of knowing that the interesting thing about 
                                                a std::vector is the elements that you get by calling size (for the summary) and [] for 
                                                the elements.  But it shouldn't be hard to come up with a generic markup to express this.
                                              </p>
                                            </li>

                                            <li>
                                              Allow the expression parser to access dynamic type/data formatter information
                                              <p>
                                                This seems like a smaller one.  The symptom is your object is Foo child of Bar, and in 
                                                the Locals view you see all the fields of Foo, but because the static type of the object 
                                                is Bar, you can't see any of the fields of Foo.  But if you could get this working, 
                                                you could hijack the mechanism to make the results of the value object summaries/synthetic 
                                                children available to expressions.  And if you can do that, you could add other properties 
                                                to an object externally (through Python or some other extension point) and then have 
                                                these also available in the expression parser.  You could use this to express invariants 
                                                for data structures, or other more advanced uses of types in the debugger.
                                              </p>
                                              <p>
                                                Another version of this is to allow access to synthetic children in the expression 
                                                parser.  Otherwise you end up in situations like:
                                              </p>
<code><pre><tt>
(lldb) print return_a_foo()
(SomeVectorLikeType) $1 = {
  [0] = 0
  [1] = 1
  [2] = 2
  [3] = 3
  [4] = 4
}
</tt></pre></code>

                                              <p>
                                                That's good but:
                                              </p>
<code><pre><tt>
(lldb) print return_a_foo()[2]
</tt></pre></code>
                                              <p>
                                              fails because the expression parser doesn't know anything about the array-like nature of 
                                              SomeVectorLikeType that it gets from the synthetic children.
                                              </p>
                                            </li>

                                            <li>
                                              Recover thread information lazily
                                              <p>
                                                LLDB stores all the user intentions for a thread in the ThreadPlans stored in 
                                                the Thread class.  That allows us to reliably implement a very natural model for
                                                users moving through a debug session.  For example, if step-over stops at a breakpoint
                                                in an function in a younger region of the stack, continue will complete the step-over
                                                rather than having to manually step out.  But that means that it is important that
                                                the Thread objects live as long as the Threads they represent.  For programs with many
                                                threads, but only one that you are debugging, that makes stepping less efficient,
                                                since now you have to fetch the thread list on every step or stepping doesn't work
                                                correctly.  This is especially an issue when the threads are provided by an
                                                Operating System plugin, where it may take non-trivial work to reconstruct the 
                                                thread list.  It would be better to fetch threads lazily but keep "unseen" threads in 
                                                a holding area, and only retire them when we know we've fetched the whole thread list
                                                and ensured they are no longer alive.
                                              </p>
                                            </li>
                                            <li>
                                              Add an extension point in the breakpoint search machinery.
                                              <p>
                                                This would allow highly customizable, algorithmic breakpoint types, like "break on every 
                                                use of some particular instruction, or instruction pattern, etc."
                                              </p>
                                            </li>
                            
                                            <li>
                                              Make Python-backed commands first class citizens
                                              <p>
                                                As it stands, Python commands have no way to advertise their options.  They are
                                                required to parse their arguments by hand.  That leads to inconsistency, and more
                                                importantly means they can't take advantage of auto-generated help and command
                                                completion.  This leaves python-backed commands feeling worse than built-in ones.
                                              </p>
                                              <p>
                                                As part of this job, it would also be great to hook automatically hook the "type" of an option value
                                                or argument (e.g. eArgTypeShlibName) to sensible default completers.  You need to be able to
                                                over-ride this in more complicated scenarios (like in "break set" where the presence of
                                                a "-s" option limits the search for completion of a "-n" option.)  But in common cases it is
                                                unnecessary busy-work to have to supply the completer AND the type.  If this worked, then 
                                                it would be easier for Python commands to also get correct completers.
                                              </p>
                                            </li>

                                            <li>
                                              Reimplement the command interpreter commands using the SB API
                                              <p>
                                                Currently, all the CommandObject::DoExecute methods are implemented 
                                                using the lldb_private API's.  That generally means that there's code
                                                that gets duplicated between the CommandObject and the SB API that does
                                                roughly the same thing.  We would reduce this code duplication, present a
                                                single coherent face to the users of lldb, and keep
                                                ourselves more honest about what we need in the SB API's if we implemented
                                                the CommandObjects::DoExecute methods using the SB API's.
                                              </p>
                                              <p>
                                                BTW, it is only the way it was much easier to develop lldb if it had a functioning
                                                command-line early on.  So we did that first, and developed the SB API's when lldb
                                                was more mature.  There's no good technical reason to have the commands use the
                                                lldb_private API's.
                                              </p>
                                            </li>

                                            <li>
                                              Documentation and better examples

                                              <p>
                                                We need to put the lldb syntax docs in the tutorial somewhere that is more easily accessible.  
                                                On suggestion is to add non-command based help to the help system, and then have a "help lldb" 
                                                or "help syntax" type command with this info.  Be nice if the non-command based help could 
                                                be hierarchical so you could make topics.
                                              </p>

                                              <p>
                                                There's a fair bit of docs about the SB API's, but it is spotty.  Some classes are 
                                                well documented in the Python "help (lldb.SBWhatever)" and some are not.
                                              </p>


                                              <p>
                                                We need more conceptual docs.  And we need more examples.  And we could provide a 
                                                clean pluggable example for using LLDB standalone from Python.  The process_events.py 
                                                is a start of this, but it just handles process events, and it is really a quick sketch 
                                                not a polished expandable proto-tool.  
                                              </p>

                                            </li>

                                            <li>
                                              Make a more accessible plugin architecture for lldb.

                                              <p>
                                                Right now, you can only use the Python or SB API's to extend an extant lldb.  You can't 
                                                implement any of the actual lldb Plugins as plugins.  That means anybody that wants to 
                                                add new Object file/Process/Language etc support has to build and distribute their own 
                                                lldb.  This is tricky because the API's the plugins use are currently not stable (and 
                                                recently have been changing quite a lot.)  We would have to define a subset of lldb_private 
                                                that you could use, and some way of telling whether the plugins were compatible with 
                                                the lldb.  But long-term, making this sort of extension possible will make lldb more 
                                                appealing for research and 3rd party uses.
                                              </p>
                                            </li>

                                            <li>
                                              Use instruction emulation to avoid the overhead of swapping trap and instruction for breakpoints

                                              <p>
                                                At present, breakpoints are implemented by inserting a trap instruction, then when the trap 
                                                is hit, replace the trap with the actual instruction and single step.  Then swap back and 
                                                continue.  This causes problems for read only text, and also means that no-stop debugging 
                                                ust either stop all threads briefly to handle this two-step or risk missing some 
                                                breakpoint hits.  If you emulated the instruction and wrote back the results, you wouldn't 
                                                have these problems, and it would also save a stop per breakpoint hit.  Since we use breakpoints 
                                                to implement stepping, this savings could be significant on slow connections.
                                              </p>
                                            </li>
                                            <li>
                                              Use the JIT to speed up conditional breakpoint evaluation

                                              <p>
                                                We already JIT and cache the conditional expressions for breakpoints for the C family of 
                                                languages, so we aren't re-compiling every time you hit the breakpoint.  And if we couldn't 
                                                IR interpret the expression, we leave the JIT'ed code in place for reuse.  But it would 
                                                be even better if we could also insert the "stop or not" decision into the code at the 
                                                breakpoint, so you would only actually stop the process when the condition was true.  
                                                Greg's idea was that if you had a conditional breakpoint set when you started the 
                                                debug session, Xcode could rebuild and insert enough no-ops that we could instrument 
                                                the breakpoint site and call the conditional expression, and only trap if the conditional was true.
                                              </p>
                                            </li>

                                            <li>
                                              Broaden the idea in "target stop-hook" to cover more events in the debugger

                                              <p>
                                                Shared library loads, command execution, User directed memory/register reads and writes 
                                                are all places where you would reasonably want to hook into the debugger.
                                              </p>
                                            </li>

                                            <li>
                                              Mock classes for testing
                                              <p>
                                                We need  "ProcessMock" and "ObjectFileMock" and the like.  These would be real
                                                plugin implementations for their underlying lldb classes, with the addition 
                                                that you can prime them from some sort of text based input files.  For classes 
                                                that manage changes over time (like process) you would need to program the state
                                                at StopPoint 0, StopPoint 1, etc.  These could then be used for testing reactions 
                                                to complex threading 
                                                problems & the like, and also for simulating hard-to-test environments (like bare 
                                                board debugging).  
                                              </p>
                                            </li>

                                            <li>
                                              A Bug-Trapper infrastructure
                                              <p>
                                                We very often have bugs that can't be reproduced locally.  So having a bug-report-trapper 
                                                that can gather enough information from the 
                                                surroundings of a bug so that we can replay the session locally would be a big help
                                                tracking down issues in this situation.  This is tricky because you 
                                                can't necessarily require folks to leak information about their code in order to 
                                                file bug reports.  So not only will you have to figure out what state to gather, 
                                                you're also going to have to anonymize it somehow.  But we very often have bugs 
                                                from people that can't reduce the problem to a simple test case and can't give us 
                                                our code, and we often just can't help them as things stand now.  Note that adding
                                                the ProcessMock would be a good first stage towards this, since you could make a ProcessMock 
                                                creator/serializer from the current lldb state.
                                              </p>
                                            </li>

                                            <li>
                                              Expression parser needs syntax for "{symbol,type} A in CU B.cpp" etc.

                                              <p>
                                                Sometimes you need to specify non-visible or ambiguous types to the expression parser.  
                                                We were planning to do $b_dot_cpp$A or something like.  You might want to specify a
                                                static in a function, in a source file, or in a shared library.  So the syntax should
                                                support all these.
                                              </p>
                                            </li>

                                            <li>
                                              Add a "testButDontAbort" style test to the UnitTest framework.

                                              <p>
                                                The way we use unittest now (maybe this is the only way it can work, I don't know) 
                                                you can't report a real failure and continue with the test.  That is appropriate 
                                                in some cases: if I'm supposed to hit breakpoint A before I evaluate an expression, 
                                                and don't hit breakpoint A, the test should fail.  But it means that if I want to 
                                                test five different expressions, I can either do it in one test, which is good 
                                                because it means I only have to fire up one process, attach to it, and get it to 
                                                a certain point.  But it also means if the first test fails, the other four don't 
                                                even get run.  So though at first we wrote a bunch of test like this, as time went 
                                                on we switched more to writing "one at a time" tests because they were more robust 
                                                against a single failure.  That makes the test suite run much more slowly.  It 
                                                would be great to add a "test_but_dont_abort" variant of the tests, then we could 
                                                gang tests that all drive to the same place and do similar things.  As an added 
                                                benefit, it would allow us to be more thorough in writing tests, since each 
                                                test would have lower costs.
                                              </p>
                                            </li>

                                            <li>
                                              Unify Watchpoint's & Breakpoints.  
                                              <p>
                                                Option handling isn't shared, and more importantly the PerformAction's have a lot 
                                                of duplicated common code, most of which works less well on the Watchpoint side.
                                              </p>
                                            </li>

                                            <li>
                                              Reverse debugging.  
                                              <p>
                                                This is kind of a holy grail, it's hard to support for complex apps (many 
                                                threads, shared memory, etc.)  But it would be SO nice to have...
                                              </p>
                                            </li>
                                            
                                            <li>
                                              Non-stop debugging.  
                                              
                                              <p>
                                                By this I mean allowing some threads in the target program to run while stopping 
                                                other threads.  This is supported in name in lldb at present, but lldb makes the 
                                                assumption "If I get a stop, I won't get another stop unless I actually run the 
                                                program." in a bunch of places so getting it to work reliably will be some a 
                                                good bit of work.  And figuring out how to present this in the UI will also be tricky.
                                              </p>
                                            </li>

                                            <li>
                                              Fix and continue.  
                                              <p>
                                                We did this in gdb without a real JIT.  The implementation shouldn't be that hard,
                                                especially if you can build the executable for fix and continue.  The tricky part is
                                                how to verify that the user can only do the kinds of fixes that are safe to do.  
                                                No changing object sizes is easy to detect, but there were many more subtle changes 
                                                (function you are fixing is on the stack...) that take more work to prevent.  
                                                And then you have to explain these conditions the user in some helpful way.
                                              </p>
                                            </li>
                                          </ol>
                </div>
          	<div class="postfooter"></div>
                </div>
      	</div>
	</div>
</div>
</body>
</html>