 IBM Kernel Hackers Respond |
 Posted by
Roblimo
on Tuesday June 18, @01:30PM
from the peek-inside-the-big-blue-monolith dept.
Dave Hansen, the IBM programmer who organized this interview (questions were posted on May 28), says, "Perhaps
I didn't make this clear enough during the call for questions, but myself
and my group are kernel programmers. But, we were able to dredge
up some responses for answers that we couldn't do ourselves. We haven't
been able to get an answer to the ViaVoice question yet, but if there is
real interest, I'll make sure that we do get some kind of answer back to
Slashdot.
IBM Kernel Hackers:
A note: we answered these questions individually, but in the interests of
Slashdot's disk space, we decided to coalesce the answers
into a single, unified one. You might say we "became one
voice". (the IBMers in the audience will get that one) These
were edited by management, but they mostly corrected our spelling
mistakes and cleaned up our dirty language :)
Remember, if you're interested in Linux on large systems or if you
have more questions, be sure to check out the LSE site, find us on LKML, or look
for us at OLS (we're
giving lots of talks).
1) Multi-CPU Scalability
by morbid
Now that Linux has been ported to run on high-end machines under
virtualization, when will we see a kernel tuned for (e.g.)
scalability to 64-128 processors natively?
IBM Kernel Hackers:
Assuming you're talking about single systems running one instance of
Linux, we are focusing on 8 way scalability this year, 16-32
scalability next year. After that, we'll do whatever the hardware
people can produce ... there aren't many 64-128 processor systems
around.
The open source community is tackling the complexities in getting
multi-cpu systems to scale well, and with that understanding also
comes a realization that sometimes entire subsystems are bottlenecks.
Major rewrites of some of these are underway in 2.5 (not just by IBM
people mind you). The scheduler is being wrestled to the ground, the
I/O subsystem is being dissected, and virtual memory implementations
are creating rhetoric worthy of the Cold War. All of these efforts
have had contributions from IBM people in Beaverton and other parts of
the Linux Technology Center.
2) OS Blending
by 2names
As Linux developers inside IBM, do you get to see the AIX source
code? If you do, are you allowed to "steal" some ideas from AIX and
implement them in Linux? If not, why not, and what's the IBM
official line?
IBM Kernel Hackers:
First of all, before any of us were allowed to contribute to Linux, we
were required to take an "Open Source Developers" class. This class
gives us the guidelines we need to participate effectively in the open
source community - both IBM guidelines and lessons learned about open
source from others in IBM.
We are definitely not allowed to cut and paste proprietary code into
any open source projects (or vice versa!). There is an IBM committee
who can and do approve the release of IBM
proprietary or patented technology, like RCU.
That covers "stealing" code, but what about ideas? We might talk to
an AIX programmer and comment we're seeing performance issues in Linux
in this area or that area and she tells us they discovered that they
really needed to profile the network routines when they saw that.
Having solved the problem once, our non-Linux peers can help steer us
without spelling it out for us, allowing us to still develop solutions
that can then be open sourced.
It's a fine line to walk, especially as an engineer who just
wants the answer :)
3) The Open Source model
by larry bagina
IBM will be using linux to help sell their hardware. Other companies
have tried this (VA Linux, which owns Slashdot, once had linux hackers
on their payroll). Obviously, IBM's hardware is in a different league
as an x86 clone, but do you have any thoughts on Open Source business
models and their validity? Once the kernel is running smoothly, will
you be disposable since the "Open Source community" can continue
development for free?
IBM Kernel Hackers:
We think the Open Source business model is more than just valid, it is
revolutionary. Linux has become a real "killer app"; the ability to run
Linux on IBM hardware is increasingly high on customer's lists. Being
able to run it doesn't really hurt AIX or VM, but not being able to run
it would cost hardware sales.
As far as our disposability: don't get too concerned on our behalf,
Linux will always have bugs and there will always be room for
improvement.
4) Getting your changes accepted?
by korpiq
Is Linus accepting your changes well? How directly do you submit
patches, and what are your experiences on the overall Linux kernel
development style?
IBM Kernel Hackers:
Linus himself is wonderful about accepting patches on
technical merit alone. He doesn't "grade" them differently if they come
from ibm.com or mit.edu. We submit patches the exact same way that
everyone else does: append the patch, mail to Linus and CC
linux-kernel. If it's good, it gets in. If it sucks, you get flamed.
However, the submission process can be more complicated than first
appears. Often, you need to figure out who is maintaining a
particular area of code, followed by talking to them to gauge if
someone else is already working on the same thing. Once you submit
your code to them and the appropriate list, (isn't always lkml..) you
may not get a response. This can be discouraging, but you have to
find out why, or just simply resubmit, over and over and over. But,
once you have a reputation, it does get easier to get quicker
responses.
Sometimes it's frustrating when you've put a lot of effort into
something that doesn't get accepted, but there's normally a good
reason for it. Even work that doesn't get accepted can influence other
people's thinking and development in the future. On the flip side you
can also just point out problems and other people fix them for you, so
in general you win more than you lose ;-)
5)linux on thinkpads
by Olinator
IMHO, IBM makes some of the best
mobile hardware out there -- one of the professors I support raves
about his ThinkPad 600, that went with him into the Israeli desert for
several months and is still running strong, no service required -- but
the linux support for that hardware has been, um, erratic at
best. Yes, we've been occasionally been able to purchase the odd model
with linux preinstalled (usually it's more expensive than the
comparable model with MicroSoft preinstalled, grr) but an awful lot of
the hardware (mini-pci modems, etc...) is rather difficult to drive
with a penguin behind the wheel. Why does IBM's linux enthusiasm fade so
quickly at the small (physical) end of the hardware scale? Is there
momentum underway to change this?
IBM Kernel Hackers:
All of the people in our group and most in the LTC have Thinkpads for
their daily development and run Linux on them (I'm writing this on one
as I sit in my apartment). There may not be as much corporate support
there as you want, but there is plenty of grass-roots support. We had
to learn all the quirks to get Linux installed and get all of the
little things working (just like you). I've always wished that we
shared more of this information, but there are usually people who are
farther ahead than we are. I've uploaded the meager information that
we put together during a meeting once. If you're curious, take a
look: http://www.sr71.net/slashdot/thinkpad/linux-desktop
People don't buy many small computers just because they will
run Linux (the geek population just isn't that large). People do,
however, blow large chunks of cash on big machines just to
run Linux. Mom-and-Pop can almost always undercut IBM on prices for
small machines, and geeks are thrifty. You don't have to sell many
million dollar machines to justify being involved in Linux development.
6) Issues with middle management
by Consul
When you were starting out as a group, did you encounter a lot of
friction and resistance from middle and/or upper management about
your wanting to work on Open Source projects for IBM? If so, what
did you do to overcome the objections and become the team you are
now? I think the answer to this would help a lot of other people
in other companies get mainstream acceptance of the idea of OSS
in corporate environments.
IBM Kernel Hackers:
The management chain from engineer up to VP has been surprisingly a
non-issue. We believe this is mostly because of the way the Linux
Technology Center was founded. You might think the LTC evolved "up"
from renegade engineers, but the truth is that our first Linux
corporate strategy in 1998 called for the creation of a team, composed
of some our best OS engineers, that would join the community to
- Learn from doing,
- Grow Linux skills
- Give back to the community
- Help make Linux better.
The LTC was founded then with full corporate backing already in place.
Boring!! The early funding story for the LTC did involve renegade
behavior and a table outside the BOFs at LinuxWorld in 1999, but
that's a different story. Our director, Dan Frye, is owed much of the
praise for the LTC's existence and smooth operation within IBM. The
critical thing here is that engineers shouldn't be fighting corporate
political battles, you need management that will go to bat for you on
those issues (and win ;-)) We have that, and they do a good job of
keeping that stuff off our backs. On the whole, our only work in this
arena is to inform the management of what we need, and to feed them
with the ammunition they need from a technical standpoint to fight
these battles for us.
7) When do you estimate Linux can surpass Solaris?
by wytcld
Solaris 9 is getting great reviews. Between the
strengths of the traditional open source community and IBM's resources, do
you see a point in the next several years where you expect Linux to surpass
Solaris in all of its core strengths? Or does Solaris have some unique values
which will allow Sun to continue to position itself to advantage, at least
for some applications? Please answer this as a technical rather than marketing
question. IBM Kernel Hackers:
We don't have a Solaris machines to back up any claim we may make, nor
do we want to stir up another epic Linux
on mainframe battle. It is safe to say, however, that
today Linux/x86 is able to outperform1
Solaris/Sparc in many areas that Sun has a long history of success.
If your core business was threatened, wouldn't you make some serious
changes?
1 I know, I know, outperform is a very vague term. Just
think price, performance, stability, etc...
8) OS/2 Developers
by reaper20
I'm one of the few people who really enjoyed the
OS/2 desktop and its features. Have any of the former OS/2 developers been
contributing to Linux?
Specifically, the user interface and accessibility
people - OS/2 was very polished - does IBM see a benefit by offering this
expertise to the GNOME/KDE projects?
If so, how does this tie into IBM's vision of Linux of the desktop, if you have one? :)
IBM Kernel Hackers:
Yes, there are a number of former OS/2 developers in the LTC including
the majority of the teams working on: JFS,
EVMS, and Print, as well
individuals contributing in the areas of networking, security, RAS,
performance and other projects. Remember, OS/2 had JFS support and
EVMS supports the OS/2 partitioning scheme.
While IBM is not actively contributing code from the OS/2 user
interface, we are supporting and sponsoring both the GNOME and the KDE
projects through our involvement in the KDE League and the GNOME
Foundation. And as you mentioned, we place a high level of importance
on accessibility and so are participating in the community efforts in
that area as well.
10) IA64
by sabre ...
Do you think that IPF64 line will see any
kind of broad industry adoption? Will it become just like rest of
the (non-embedded) processor architectures designed since the x86
-- constantly fighting for 5% of the market? Do you think the AMD
Hammer architecture will be a meaningful player in the field?
IBM Kernel Hackers:
Quite possibly, never underestimate the importance of being able to
run the huge installed base of ia32 apps natively, and at high speed.
But IA64 has lots of industry backing as well. The good news is that
Linux runs well on both, so we the community don't have to choose.
The market will do that for us.
Additional questions and answers:
What features do you find linux most lacking in? (If we don't
examine our weaknesses, we will be crippled)
Linux on the desktop still doesn't really cut it for some of us
(though we do use it). Applications are not nearly as robust as they
should be, and though we are perfectly capable of configuring X, we'd
rather spend the time coding. Though it's fun to throw stones at
Windows and the Linux OS is more stable than the Windows OS, as a
whole desktop package with the apps, installation, usability and
everything rolled together, Linux is not always preferable.
There are thorns in our side daily because of the lack of debugging
and profiling ability in the kernel. We're always patching kernels
for kernprof or lockmeter and porting them around to new kernel
versions. Although Linus has pretty much said that debuggers are for
sissies, the built-in facilities are much better than they were during
the old days (think readprofile). So, there are advances being
made.
...I'm not surprised that your responses have to be
vetted by management. But, I'd love to know what guidelines IBM has
for hackers' interaction with the rest of the GNU/Linux/Internet
community. Are you allowed to criticize IBM management, or other IBM
products, for example?
This is the Internet. We are hackers. Our management has been great
allowing us to resolve many of our own problems involving certain
email systems and desktop OS rules. Working in the kernel group of the
LTC we have free reign to do our work on the kernel in the Open Source
community.
There's no day-to-day vetting of anything we post or say, they trust
us to be sensible. We would not say "IBM product X sucks, and you
should buy competitor's product Y instead" in a public forum, but if
we don't think something works well, I'm not going to endorse it
either. We're engineers who get paid to work on Linux by IBM, not IBM
corporate drones ;-)
From the brief bios, and Sequent pedigree, it looks like there
is a lot of focus on high-end features like NUMA, async I/O and the like.
Other commercial organizations, notably SGI, are also putting forth
effort in those areas. There is actually quite a bit of overlap.
Since these are "open source" projects, do you collaborate with your
traditional "enemies" such as SGI and Sun on Linux? What is your
management's attitude toward that type of collaboration? If not, do
you "look" at the work \ the others are doing in comparison to what
you are doing?
We have been working smoothly with engineers from HP, Intel, SGI and
many other companies through the Linux Scalability Effort Open Source
Project. Whatever legal issues there might be within each company, it
appears to me that the engineers who are working on open source are
allowed to do their work with no problems. Hanna runs the bi-weekly
LSE Conference Call and can say the biggest percentage of attendees
are from either IBM, SGI or Intel every time. This is nice, but we want
more members from the Open Source community to join:
(http://lse.sf.net/mtg).
Management doesn't really care too much who actually writes the patch
at the end of the day, they want to see Linux work well in their focus
areas. Persuading other (external) developers of the correct approach
or solution to a problem is just as important a part of our jobs as
writing code.
Why isn't IBM making more of an effort to recruit developers
directly from the Linux community, as opposed to hiring people who
have very little if any working familiarity with the platform?
IBM has hired lots of existing Linux developers such as Rusty Russell,
Greg Kroah-Hartman, and Ted Ts'o. There are also others that post to
LKML and don't even use their IBM email addresses because they were
firmly established community members (with those email addresses)
before they were hired. We do have a site where you can view
many of our patches, or a list
of developers. Keep in mind that there are still developers who
don't submit patches here.
What are your opinions regarding the shrinking number of women in the
industry? (actually I believe the numbers are rising again in schools)
This is a tough field and many young women are discouraged from
sticking it out through all of the math and science classes, as are
many young men. Companies, like IBM, help by hiring bright women who
move up the technical chain. This shows the less experienced that
there is a future for women in engineering.
Shrinking? The numbers seem to be increasing based on what we see at
work every day. IBM is active in programs that introduce young women
to engineering in an attempt to get them interested in pursing careers
in engineering. An example of these programs is Camp EXITE, check
this site out for more info:
http://www.ibm.com/ibm/ibmgives/grant/education/camp.shtml
An interesting interview regarding the number of women in industry is
available at: http://www.nspe.org/etweb/16-02viewpoint.asp
Questions Rick Lindsley liked that didn't make the top 10,
plus answers:
Best way into the Professional Linux world?
As many people here, I am a huge Linux fan, but I am so much so
that I am trying to figure out how to get into the professional
Linux world when I graduate.
I attend Clemson University and am in the Computer
Information System (CS + business) program (and
doubled in Political Science). My goal is to become a
Linux sys admin, or perhaps some other Linux guru type job. The
work that IBM is doing with Linux is also very appealing to me.
So, how did you get your job, and what would you recommend as the
path to follow for us geeks just getting started in the
professional world as to how to get into Linux? How can I become as
entrenched with Linux as the professionals at IBM? I have had two
internships (not with IBM, nor with Linux, but with other CS
stuff), but how can I get an entry-level job in a Linux intensive
environment like IBM? How can said job lead me into a career where
I can be deeply involved in the Linux world?
Rick:
First: I've done recruiting at a "significant Big 10 university
whose mascot's name is Bucky" so let me tell you what I look for in
a college candidate.
Knowledgable -- your resume should reflect what you know, but don't
puff it. Just because they make me dress up when I'm on campus
doesn't mean I can't tell perl from shell scripting. Accentuate
your strong points. You gain points for knowledge, but you lose
them for lying or "overstating."
Communicative -- a person who cannot talk about what they know might
as well know nothing. Seriously consider taking a public speaking
course your junior or senior year. Also: it's ok to say "I don't
know."
Grade point -- Personally, I really don't care so much about your
GPA as you might think. Unfortunately, you will be judged by
it by far too many people, right or wrong. So if you're not
3.9 or 4.0, you might be ready to spin it a bit. "Yes, it's
3.2, but I've buckled down and have 3.84 in the last three
semesters." "Yes it's 3.1, but you'll note it's 3.6 on courses
in my major." Don't get surreal but make that number say
something good about you.
Work experience -- you get a big edge for doing something other
than a teaching assistant. Internships, co-ops, and summer
jobs can help you more than you think in the end.
Second, how did I get my job at IBM? Luck, in part. Right place,
right time. Sometimes it really does work for you. Along with that
luck, though, was the fact that I'd established a reputation as a
smart coder and a fast learner. While I knew far less about Linux
then than I know now, that reputation made managers believe that
"coming up to speed" would not be a problem, and they judged
right. Your reputation, as reported by your colleagues and not
yourself, will be your greatest ally (or enemy.) This is never
more true than in the Linux community.
Third, advancing? Once you get your foot in the door, work at
interacting. Nobody really advances very far without interaction.
At first this is with your cubie neighbor or office mate, but
pretty soon it's chatting with people down the hall, and then in
other projects. Eventually, you have opportunities to help
organize informal seminars with the local user's group, and then
it's helping out with conferences, and then you're writing papers,
and chairing sessions, and before you know it you're standing
puzzled in front of a thousand people, wondering how troubled their
life must be that they would want to listen to <em>you</em> speak.
Dave Hansen's answer -
Purdue University's Computer Science program. I went
to one of the CS job fairs where someone in the large IBM booth saw
"Linux" on my resume. I handed my resume off, had a nice chat, and
got a sit-down interview a couple of days later. That was followed
soon by a plant trip and a job offer. The moral of the story: if you
want a Linux job, put Linux on your resume! Make it bold. Make it
half the page if that is really want you want to do. Most
importantly, you have to learn to walk the walk before you can talk
the talk. Engineers usually have better BS detectors than most people
and you won't fool them for long.
Advancing - This is probably evident to anyone who has gone through an
engineering program at a large school, but the most successful
engineers are those who can teach others. You'll notice that there
are lots of brilliant engineers and lots of teachers, but those who
can do both are a rarity. Learn as much as you can from your
colleagues then share as much as you can. The more people who know
your name and come to you for help, the more visible you are. There
is probably a fine line between getting noticed and being annoying and
I have the feeling that a Slashdot interview may be WAY beyond the
line :)
| 
