An object database (more correctly referred to as ODBMS or OODBMS for Object Database Management System) is a DBMS that stores objects as opposed to rows or tuples in respectively a SQL or a RDBMS (Relational Database Management System).
It is most often used in the case of C++ and Java programmers that do not wish to deal with the impedance mismatch of going from an object oriented (OO) programming language to a database query language like SQL programming language that is the current standard. Developers prefer to be able to persist an object without having to go through a paradigm shift. Also missing from SQL is the concept of polymorphism, which is central to OO design, thus causing headaches when mapping from OO code to SQL.
Of course this has advantages and disadvantages. The ability to stay with an OO paradigm does great things for productivity. However, the SQL DBMSs is a mature and proven one that has had decades of development and testing.
Certain benchmarks between ODBMS and SQL DBMSs have shown that ODBMS can be clearly superior. One of the main reasons is that ODBMS do not use joins to associate objects but references which are usually implemented as pointers. In SQL, a join would in most cases minimally require a search through a B-tree index. In general, navigation in an ODBMS is via traversing references, whereas in SQL data is joined in an ad-hoc fashion (which is better for arbitrary queries).
The successful market segments for ODBMS seem to be in telecommunications, high energy physics and subsets of financial services. The things that work against ODBMS seem to be the lack of interoperability with a great number of tools/features that are taken for granted in the SQL world including but not limited to industry standard connectivity, reporting tools, OLAP tools and backup and recovery standards. Additionally, ODBMS lacks a formal mathematical foundation, unlike the relational model, which rests upon the firm and well-understood mathematical basis of the relational calculus.
It must be noted that all SQL limitations come from it violating the principles of the relational model. A proper implementation, such as Alphora Dataphor, provides all the benefits of OO DBMSs without their drawbacks.
The Object Database Management Group did come up with a industry standard called ODMG 2.0 but it failed to gain acceptance with the ODBMS vendors mostly opting for proprietary features and extensions instead of attempting to grow the ODBMS pie by standards compliance and competing on implementations.
As an industry, ODBMS are a lost opportunity to revolutionize software development. Instead, there are more solutions out there providing wrongly so-called object-relational mapping abilities -- actually object-SQL mappings -- than there are ODBMS. There are many design patterns for designing object-SQL mappings (it is a non-trivial task; see notes to this effect in database).