Hi,
I have a request for an additional feature to be included
in the recent hardware breakpoints work soon to be delivered
in kernel 2.6.33.
Frederic Weisbecker, Prasad Krishnan and I have shared an ongoing
private discussion on this topic. Frederic recommended that we take
the discussion to a wider audience. Hence this email. I would like
to thank both
of those two engineers for their patience, quick response,
and help in this process. They've been stellar.
I work for a company which would like to make heavy use
of the new HW breakpoints perf_event work. Essentially,
we need to be able to do the following:
1) When executed, a user-land application must be able to program 4
pinned hardware
breakpoint registers. I need 1 byte granularity (address length
specificity) and the ability
to set RWX event triggers.
2) All calls to clone() or fork() will propagate the
debug register settings from parent to child(ren).
3) When a breakpoint is triggered, the application
thread currently running which triggered the breakpoint
immediately stops execution and is sent a SIGTRAP.
4) The thread transitions from the PC that triggered the breakpoint to
the signal handler for SIGTRAP.
5) The signal handler does some work. (This "work" is outside the
scope of my request, but you may have some insights. I need to be
able to change the PC and nPC for the thread that triggered a
breakpoint such that when it
returns from the signal handler it doesn't return to the instruction
that triggered the breakpoint but to the one after it. If I were
using ptrace(8), I would just have the parent
process use the ptrace(8) syscall to modify the PC and
nPC of the child. I'm not using ptrace(8).)
6) The signal handler returns and the thread returns to normal
execution at the new
PC and nPC.
>From my discussions with Frederic and Prasad, I know
that requirements (1) and (2), as described above,
are already being met by the current work. I can use
the perf_event_attr structure and the perf_event syscall
to program breakpoints for a thread in exactly
the way I've specified AND be able to have debug
settings inherited from parent to child in the case of a
fork() or clone().
However, there's nothing yet in place to allow a
signal to be sent to the thread when a breakpoint
has been hit. Put another way, there's nothing here
which affects execution flow of the user-app when the CPU
traps due to a HW breakpoint.
Currently, the perf events infrastructure allows me
to mmap() a block of memory and to poll() on it, waiting
and watching for events to be described in that mmap'd
buffer. That will not be sufficient for what we need
to do. We need to have the ability to specify that
execution of the thread should be interrupted, just as it
would under ptrace(), and have a signal be delivered.
Delivery of the signal must be received and processed
by the application before the thread will be allowed to
proceed to the nPC after the PC which caused the
HW breakpoint event.
Is this possible? Can we architect this feature into
the perf_events infrastructure?
At first glance, it would seem that this is a fairly
easy thing to do. Right now, these same HW breakpoints
triggers under ptrace() merely call a previously registered
user handler which modifies the debug registers to
record the event and to propagate a signal to the child
process being traced. We want to do something similar
w/o using ptrace. Perhaps it is as simple as providing a
perf_event_attr setting which indicates that SIGTRAP signals
should be sent to the thread which triggered the breakpoint
exception.
Thanks in advance for your help,
JP
Joshua Pincus <[email protected]> writes:
> I have a request for an additional feature to be included
> in the recent hardware breakpoints work soon to be delivered
> in kernel 2.6.33.
Sounds to me like the existing ptrace based interface
can practically all you want
(except that the "parent signal" would be wait and for
fork/exec you have to explicitely attach)
-Andi
--
[email protected] -- Speaking for myself only.
Hi,
On Wed, Jan 13, 2010 at 6:15 PM, Andi Kleen <[email protected]> wrote:
> Joshua Pincus <[email protected]> writes:
>
>> I have a request for an additional feature to be included
>> in the recent hardware breakpoints work soon to be delivered
>> in kernel 2.6.33.
>
> Sounds to me like the existing ptrace based interface
> can practically all you want
>
> (except that the "parent signal" would be wait and for
> fork/exec you have to explicitely attach)
We would like to avoid using ptrace at all costs.
It requires us to have a parent thread running
which monitors all the others. It's not clear that
the wait() call by the parent doesn't mask a barrage
of signals from various threads and the performance
penalty is huge in multi-threaded apps.
If we could get this functionality working w/o ptrace,
we'd be very, very happy and grateful.
>
> -Andi
>
Thanks,
JP
--
On Wed, Jan 13, 2010 at 6:15 PM, Andi Kleen <[email protected]> wrote:
> Joshua Pincus <[email protected]> writes:
>
>> I have a request for an additional feature to be included
>> in the recent hardware breakpoints work soon to be delivered
>> in kernel 2.6.33.
>
> Sounds to me like the existing ptrace based interface
> can practically all you want
>
> (except that the "parent signal" would be wait and for
> fork/exec you have to explicitely attach)
>
> -Andi
>
> --
> [email protected] -- Speaking for myself only.
>
(Adding Ingo in Cc)
On Wed, Jan 13, 2010 at 05:45:55PM -0800, Joshua Pincus wrote:
> Hi,
>
> I have a request for an additional feature to be included
> in the recent hardware breakpoints work soon to be delivered
> in kernel 2.6.33.
>
> Frederic Weisbecker, Prasad Krishnan and I have shared an ongoing
> private discussion on this topic. Frederic recommended that we take
> the discussion to a wider audience. Hence this email. I would like
> to thank both
> of those two engineers for their patience, quick response,
> and help in this process. They've been stellar.
>
> I work for a company which would like to make heavy use
> of the new HW breakpoints perf_event work. Essentially,
> we need to be able to do the following:
>
> 1) When executed, a user-land application must be able to program 4
> pinned hardware
> breakpoint registers. I need 1 byte granularity (address length
> specificity) and the ability
> to set RWX event triggers.
>
> 2) All calls to clone() or fork() will propagate the
> debug register settings from parent to child(ren).
>
> 3) When a breakpoint is triggered, the application
> thread currently running which triggered the breakpoint
> immediately stops execution and is sent a SIGTRAP.
>
> 4) The thread transitions from the PC that triggered the breakpoint to
> the signal handler for SIGTRAP.
>
> 5) The signal handler does some work. (This "work" is outside the
> scope of my request, but you may have some insights. I need to be
> able to change the PC and nPC for the thread that triggered a
> breakpoint such that when it
> returns from the signal handler it doesn't return to the instruction
> that triggered the breakpoint but to the one after it. If I were
> using ptrace(8), I would just have the parent
> process use the ptrace(8) syscall to modify the PC and
> nPC of the child. I'm not using ptrace(8).)
>
> 6) The signal handler returns and the thread returns to normal
> execution at the new
> PC and nPC.
>
> From my discussions with Frederic and Prasad, I know
> that requirements (1) and (2), as described above,
> are already being met by the current work. I can use
> the perf_event_attr structure and the perf_event syscall
> to program breakpoints for a thread in exactly
> the way I've specified AND be able to have debug
> settings inherited from parent to child in the case of a
> fork() or clone().
>
> However, there's nothing yet in place to allow a
> signal to be sent to the thread when a breakpoint
> has been hit. Put another way, there's nothing here
> which affects execution flow of the user-app when the CPU
> traps due to a HW breakpoint.
>
> Currently, the perf events infrastructure allows me
> to mmap() a block of memory and to poll() on it, waiting
> and watching for events to be described in that mmap'd
> buffer. That will not be sufficient for what we need
> to do. We need to have the ability to specify that
> execution of the thread should be interrupted, just as it
> would under ptrace(), and have a signal be delivered.
> Delivery of the signal must be received and processed
> by the application before the thread will be allowed to
> proceed to the nPC after the PC which caused the
> HW breakpoint event.
>
> Is this possible? Can we architect this feature into
> the perf_events infrastructure?
>
> At first glance, it would seem that this is a fairly
> easy thing to do. Right now, these same HW breakpoints
> triggers under ptrace() merely call a previously registered
> user handler which modifies the debug registers to
> record the event and to propagate a signal to the child
> process being traced. We want to do something similar
> w/o using ptrace. Perhaps it is as simple as providing a
> perf_event_attr setting which indicates that SIGTRAP signals
> should be sent to the thread which triggered the breakpoint
> exception.
>
> Thanks in advance for your help,
> JP
On Wed, 2010-01-13 at 17:45 -0800, Joshua Pincus wrote:
> However, there's nothing yet in place to allow a
> signal to be sent to the thread when a breakpoint
> has been hit. Put another way, there's nothing here
> which affects execution flow of the user-app when the CPU
> traps due to a HW breakpoint.
>
> Currently, the perf events infrastructure allows me
> to mmap() a block of memory and to poll() on it, waiting
> and watching for events to be described in that mmap'd
> buffer. That will not be sufficient for what we need
> to do. We need to have the ability to specify that
> execution of the thread should be interrupted, just as it
> would under ptrace(), and have a signal be delivered.
> Delivery of the signal must be received and processed
> by the application before the thread will be allowed to
> proceed to the nPC after the PC which caused the
> HW breakpoint event.
>
> Is this possible? Can we architect this feature into
> the perf_events infrastructure?
We have fnctl() managing signal support, but that will only interrupt
the observing thread, not the threads being observed (unless they are
one and the same -- but using inheritance precludes that from being
true).
I'm not sure I'm willing to go in this direction with perf, the idea is
to have a minimum impact on the observed threads, explicitly sending
them signals and disturbing their execution goes against this.
> We would like to avoid using ptrace at all costs.
> It requires us to have a parent thread running
> which monitors all the others. It's not clear that
> the wait() call by the parent doesn't mask a barrage
> of signals from various threads and the performance
mask? It'll report them. You expect to have so
many signals that this would be a problem?
-Andi
Hi Peter,
On Thu, Jan 14, 2010 at 12:44 AM, Peter Zijlstra
> We have fnctl() managing signal support, but that will only interrupt
> the observing thread, not the threads being observed (unless they are
> one and the same -- but using inheritance precludes that from being
> true).
>
> I'm not sure I'm willing to go in this direction with perf, the idea is
> to have a minimum impact on the observed threads, explicitly sending
> them signals and disturbing their execution goes against this.
I understand your qualms.
I'm not asking for this feature to be implemented as the
default. I'm asking that it be an option for someone like me to use.
We have a very strong
and compelling reason for doing what I've requested
vis a vis observed threads. I am sure we can come up
with other compelling reasons for doing this, not the least
of which is that this functionality has been provided
in Microsoft Windows for a decade or more and has
proven immensely useful.
For instance, one could use this as a realtime watchdog
without requiring extra observing threads to do the
signal processing work. Or having observed threads
use the signal handler to tally up events.
Without the ability for observed threads to be interrupted
in this way when they hit breakpoints, the new
perf_event work is not going to be useful to us in its
current form.
Thanks,
JP
>
>
Hi,
On Thu, Jan 14, 2010 at 1:23 AM, Andi Kleen <[email protected]> wrote:
>> We would like to avoid using ptrace at all costs.
>> It requires us to have a parent thread running
>> which monitors all the others. It's not clear that
>> the wait() call by the parent doesn't mask a barrage
>> of signals from various threads and the performance
>
> mask? It'll report them. You expect to have so
> many signals that this would be a problem?
Yes. We expect to see a zillion of them.
> -Andi
>
Thanks,
JP
On Thu, Jan 14, 2010 at 10:03:27AM -0800, Joshua Pincus wrote:
> Hi,
>
> On Thu, Jan 14, 2010 at 1:23 AM, Andi Kleen <[email protected]> wrote:
> >> We would like to avoid using ptrace at all costs.
> >> It requires us to have a parent thread running
> >> which monitors all the others. ?It's not clear that
> >> the wait() call by the parent doesn't mask a barrage
> >> of signals from various threads and the performance
> >
> > mask? It'll report them. You expect to have so
> > many signals that this would be a problem?
>
> Yes. We expect to see a zillion of them.
I don't quite understand what signals are masked here,
actually I'm not sure what is the true problem with ptrace.
Is it because a breakpoint in a thread is going to stop
all thread in the process until the parent handles the
signal?
Anyway, although I first suggested extending perf, with
more thoughts I now agree that perf should keep doing what
it does currently (profiling) and not trying to become an
messy mix of a profiler, debugger, etc...
This is not its role.
But it can certainly be used by a debugging facility.
What about extending ptrace to support a new type of
breakpoint debugging interface?
On Mon, Jan 18, 2010 at 12:04:07PM +0100, Frederic Weisbecker wrote:
> I don't quite understand what signals are masked here,
ptrace reports all signals to the tracer and only
delivers them on PTRACE_CONT. "Masking" is probably
not the right term for it.
This could be fixed relatively easily with a new signal
mask ptrace option I guess.
> actually I'm not sure what is the true problem with ptrace.
> Is it because a breakpoint in a thread is going to stop
> all thread in the process until the parent handles the
> signal?
I didn't think ptrace stopped all threads by its own,
it just stops the current one and also only until
the problem is reported. And for a break point you
typically want (in fact need to) to stop until it is handled.
>
> Anyway, although I first suggested extending perf, with
> more thoughts I now agree that perf should keep doing what
> it does currently (profiling) and not trying to become an
> messy mix of a profiler, debugger, etc...
I find it surprising that you say that -- my
impression is that whatever gets proposed recently:
error injection, testing, error handling, any other event
someone proposed perf as the answer.
"If all I got is a hammer everything starts to look like a thumb"
But yes I fully agree with you.
> What about extending ptrace to support a new type of
> breakpoint debugging interface?
There's this utrace interface, but it seems to be more
a in kernel layer with some "we can't make up our mind what
the interface to the user looks like" and a lot of
overengineering in interfaces thrown in.
ptrace in its current form is somewhat messy, but for
me it seems there isn't anything better. So yes extending
it would seem like a good idea.
A good starting point might be the debug store interfaces
that got recently added to ptrace.
-Andi
--
[email protected] -- Speaking for myself only.
Frederic Weisbecker <[email protected]> writes:
> [...]
> What about extending ptrace to support a new type of
> breakpoint debugging interface?
This is the sort of reason why the utrace-gdbstub prototype was
constructed. It should allow in-kernel implementation of the
multithreaded gdb extensions. (By the way, it can already use uprobes
rather than userspace-managed breakpoints.)
- FChE
On Wed, Jan 13, 2010 at 05:45:55PM -0800, Joshua Pincus wrote:
<snipped>
> 1) When executed, a user-land application must be able to program 4
> pinned hardware
> breakpoint registers. I need 1 byte granularity (address length
> specificity) and the ability
> to set RWX event triggers.
>
attr.pinned = 1;
attr.bp_len = HW_BREAKPOINT_LEN_1;
attr.bp_type = HW_BREAKPOINT_W | HW_BREAKPOINT_R;
(HW_BREAKPOINT_X is allowed only for user-space requests in x86, at the
moment).
> 2) All calls to clone() or fork() will propagate the
> debug register settings from parent to child(ren).
>
attr.inherit = 1;
When used with the proposed new interface:
register_user_hbp_by_pid(struct perf_event_attr * attr,
perf_overflow_handler_t triggered, pid_t pid)
(LKML reference:[email protected])
the said request is propagated to all new threads of the process and
any child processes (and its threads).
> 3) When a breakpoint is triggered, the application
> thread currently running which triggered the breakpoint
> immediately stops execution and is sent a SIGTRAP.
>
Signal generation happens by default for all user-space breakpoint
requests (and not just for ptrace requests).
> 4) The thread transitions from the PC that triggered the breakpoint to
> the signal handler for SIGTRAP.
>
> 5) The signal handler does some work. (This "work" is outside the
> scope of my request, but you may have some insights. I need to be
> able to change the PC and nPC for the thread that triggered a
> breakpoint such that when it
> returns from the signal handler it doesn't return to the instruction
> that triggered the breakpoint but to the one after it. If I were
> using ptrace(8), I would just have the parent
> process use the ptrace(8) syscall to modify the PC and
> nPC of the child. I'm not using ptrace(8).)
>
I don't see a need to do this explicitly (atleast on x86 and hopefully
for PPC64 too). Breakpoint exceptions due to data accesses are mostly
triggered after the memory access is performed. In other words, the
instruction causing the memory access is completed following which the
breakpoint exception is raised (atleast on x86). In turn, this implies
that the signal is delivered/handled after "PC" and when the execution
returns to the original thread, it resumes from the new PC (I'm not
sure nPC referred to here actually means).
> 6) The signal handler returns and the thread returns to normal
> execution at the new
> PC and nPC.
>
It is possible to achieve all the above requirements (using the
above-suggested methods) in a slightly convoluted way i.e. by using
a kernel module that makes user-space breakpoint requests by directly
invoking the hw-breakpoint API (and not through ptrace or perf syscalls).
Such a kernel module can receive parameters namely PID, user-space address
(and breakpoint attributes such as length, etc) through various ways -
as kernel module parameters or through proc, sys, debugfs interfaces.
Thanks,
K.Prasad
On Mon, Jan 18, 2010 at 12:35:59PM +0100, Andi Kleen wrote:
> On Mon, Jan 18, 2010 at 12:04:07PM +0100, Frederic Weisbecker wrote:
> > I don't quite understand what signals are masked here,
>
> ptrace reports all signals to the tracer and only
> delivers them on PTRACE_CONT. "Masking" is probably
> not the right term for it.
Ah, ok.
> > actually I'm not sure what is the true problem with ptrace.
> > Is it because a breakpoint in a thread is going to stop
> > all thread in the process until the parent handles the
> > signal?
>
> I didn't think ptrace stopped all threads by its own,
> it just stops the current one and also only until
> the problem is reported. And for a break point you
> typically want (in fact need to) to stop until it is handled.
Yeah, ok.
> > Anyway, although I first suggested extending perf, with
> > more thoughts I now agree that perf should keep doing what
> > it does currently (profiling) and not trying to become an
> > messy mix of a profiler, debugger, etc...
>
> I find it surprising that you say that -- my
> impression is that whatever gets proposed recently:
> error injection, testing, error handling, any other event
> someone proposed perf as the answer.
Event injection is not off boundaries. This is still
in the topic of profiling/tracing.
Not sure what you mean. My point was that perf should
keep being an event performance observation tool, and
that controlling the execution flow of a process is not
the role of perf.
>
> > What about extending ptrace to support a new type of
> > breakpoint debugging interface?
>
> There's this utrace interface, but it seems to be more
> a in kernel layer with some "we can't make up our mind what
> the interface to the user looks like" and a lot of
> overengineering in interfaces thrown in.
>
> ptrace in its current form is somewhat messy, but for
> me it seems there isn't anything better. So yes extending
> it would seem like a good idea.
>
> A good starting point might be the debug store interfaces
> that got recently added to ptrace.
I need to look at this then. Thanks.
On Mon, Jan 18, 2010 at 11:33:46AM -0500, Frank Ch. Eigler wrote:
> Frederic Weisbecker <[email protected]> writes:
>
> > [...]
> > What about extending ptrace to support a new type of
> > breakpoint debugging interface?
>
> This is the sort of reason why the utrace-gdbstub prototype was
> constructed. It should allow in-kernel implementation of the
> multithreaded gdb extensions. (By the way, it can already use uprobes
> rather than userspace-managed breakpoints.)
Can utrace somehow meet Joshua's needs? I'm not sure what kind
of interface it can offer for that. The fact is I don't know
very well utrace :) Do you plan a resubmission soon?
Thanks.
On Mon, Jan 18, 2010 at 11:15:05PM +0530, K.Prasad wrote:
> Signal generation happens by default for all user-space breakpoint
> requests (and not just for ptrace requests).
Ah? That's not a good thing. We should be able to
set userspace breakpoints later on perf without
generating signals.
Hi -
> > > [...]
> > > What about extending ptrace to support a new type of
> > > breakpoint debugging interface?
> >
> > This is the sort of reason why the utrace-gdbstub prototype was
> > constructed. It should allow in-kernel implementation of the
> > multithreaded gdb extensions. (By the way, it can already use uprobes
> > rather than userspace-managed breakpoints.)
>
> Can utrace somehow meet Joshua's needs?
Not directly, I'm afraid. I jumped in at a late stage of the thread
that got a bit away from Joshua's original note
(http://lkml.org/lkml/2010/1/13/490). OTOH, it could be made to work
a few different ways.
One is a systemtap or hand-written module that maps selected
perf-event hits in the kernel to an application SIGTRAP.
Another is using the gdbstub, extended with gdb watchpoint support (Z*
packets), which would tie into the hw-breakpoint system directly.
Joshua's application would manage the debug registers by means of a
userspace supervisor process sending the appropriate Z* packets to the
gdbstub, and otherwise letting the program run. When a watchpoint
fires, the supervisor process can instruct gdbstub to send a SIGTRAP
to the application. In this scenario, the perf syscall / subsystem is
not used at all.
> I'm not sure what kind of interface it can offer for that. The fact
> is I don't know very well utrace :)
That's ok. utrace is an in-kernel API for process management. ptrace
and the RFC gdbstub are two possible userspace interfaces tuned for
third-party debugging.
> Do you plan a resubmission soon?
Utrace core has been resubmitted at the end of December
(http://lkml.org/lkml/2009/12/17/466), with no further comments
received. I hope it gets plopped into linux-next ASAP and merged next
time. The gdbstub was an RFC only at this stage, but if other people
get excited about it, we're happy to spiff it up for proposed merging.
- FChE
On Tue, Jan 19, 2010 at 10:12:10AM -0500, Frank Ch. Eigler wrote:
> Hi -
>
> > > > [...]
> > > > What about extending ptrace to support a new type of
> > > > breakpoint debugging interface?
> > >
> > > This is the sort of reason why the utrace-gdbstub prototype was
> > > constructed. It should allow in-kernel implementation of the
> > > multithreaded gdb extensions. (By the way, it can already use uprobes
> > > rather than userspace-managed breakpoints.)
> >
> > Can utrace somehow meet Joshua's needs?
>
> Not directly, I'm afraid. I jumped in at a late stage of the thread
> that got a bit away from Joshua's original note
> (http://lkml.org/lkml/2010/1/13/490). OTOH, it could be made to work
> a few different ways.
>
> One is a systemtap or hand-written module that maps selected
> perf-event hits in the kernel to an application SIGTRAP.
Yeah would work.
But I rather hope we can extend ptrace interface to handle
such new needs instead (ie: having a more scalable breakpoint
interface support by ptrace).
> Another is using the gdbstub, extended with gdb watchpoint support (Z*
> packets), which would tie into the hw-breakpoint system directly.
> Joshua's application would manage the debug registers by means of a
> userspace supervisor process sending the appropriate Z* packets to the
> gdbstub, and otherwise letting the program run. When a watchpoint
> fires, the supervisor process can instruct gdbstub to send a SIGTRAP
> to the application. In this scenario, the perf syscall / subsystem is
> not used at all.
Is this gdbstub an interface to utrace?
This: http://lwn.net/Articles/364268/ ?
>
> > I'm not sure what kind of interface it can offer for that. The fact
> > is I don't know very well utrace :)
>
> That's ok. utrace is an in-kernel API for process management. ptrace
> and the RFC gdbstub are two possible userspace interfaces tuned for
> third-party debugging.
Ok.
>
> > Do you plan a resubmission soon?
>
> Utrace core has been resubmitted at the end of December
> (http://lkml.org/lkml/2009/12/17/466), with no further comments
> received.
Hmm, there has been deep review from Peter, IIRC.
> I hope it gets plopped into linux-next ASAP and merged next
> time. The gdbstub was an RFC only at this stage, but if other people
> get excited about it, we're happy to spiff it up for proposed merging.
Ok.
Thanks.
On Tue, 2010-01-19 at 17:21 +0100, Frederic Weisbecker wrote:
>
>
> Hmm, there has been deep review from Peter, IIRC.
I did provide some feedback on an earlier version, didn't get around to
look at the latest. Also I didn't cover all of it.
Hi -
On Tue, Jan 19, 2010 at 05:21:02PM +0100, Frederic Weisbecker wrote:
> [...]
> But I rather hope we can extend ptrace interface to handle such new
> needs instead (ie: having a more scalable breakpoint interface
> support by ptrace).
According to its maintainer, ptrace per se appears to be not well
suited for extensions that affect control flow, but maybe.
(http://www.mail-archive.com/[email protected]/msg02276.html)
> > Another is using the gdbstub, extended with gdb watchpoint support (Z*
> > packets), which would tie into the hw-breakpoint system directly.
> > [...]
> Is this gdbstub an interface to utrace?
> This: http://lwn.net/Articles/364268/ ?
Yes, but I wouldn't think of it that way ("an interface to utrace").
Yes, it uses utrace, but that's an implementation detail. To
userspace it presents gdb's existing wire protocol for debugging
processes.
> > > Do you plan a resubmission soon?
> >
> > Utrace core has been resubmitted at the end of December
> > (http://lkml.org/lkml/2009/12/17/466), with no further comments
> > received.
>
> Hmm, there has been deep review from Peter, IIRC.
I haven't seen any after that particular resubmission. Rather, there
has been lots of discussion lately about *uprobes*, which is a
separate & optional process breakpoint management layer that happens
to use utrace and happens to be used by systemtap and the gdbstub.
- FChE